Am I worrying too much???

When I was with my ex, we TTC with no results. After we broke up, he got married almost instantly. His wife is now pregnant, and he has two kids from a previous relationship! My SO and I have recently began trying and have been for only 1 month. I may be getting ahead of myself, but infertility has crossed my mind more than once. I'm terrified of going to the doctor to see. If the results came back that I could not mother my own children, it would be a fate worse than death. I don't want to stress it, but it's always in the back of my mind. Any advice on how to not worry so much????