Frustrated

We've been TTC for 6 months now. Everyone keeps asking when we'll have a baby, or our each set of parents will ask when they'll be grandparents. I want a baby and so does my husband. He keeps mentioning that his friends wives at work are getting pregnant. I feel bad. I know I shouldn't because it will happen when it happens. But I can't help and feel like I'm letting him down. He knows I have endometriosis and that it might never happen for us but it still makes me feel bad.