Paranoid

I'm 32 and my husband is 46. We have been TTC since January. He has 3 grown up children and jokes that he never had any problems getting his ex pregnant. His brother also boasts that he got his partner pregnant within weeks of trying. I am now starting to get paranoid about the matter. I keep telling myself that if I stress it will prolong the matter. I have always wanted children and I try to pass it off with a "what will be will be" attitude but sometimes it's not so simple. Just wondering if anyone else feels pressured at times and starts to think that it's their fault