Only been pregnant by 1 man

I am probably the only 1 this has happened to but ill be 30 in less than a month. I have 2 daughters by the same father. We havent been together in 5 yrs. & we dont plan on ever being together again we just are not good together but we are doing are best co parenting. Im in a new relationship almost 2 yrs together now & we have been trying to conceive with no luck. The problem that I have is that I have never been pregnant by another man besides my kids dad even though I have had other relationships. Before my kids dad I thought I couldnt have kids because of previous encounters with no luck. But It happened with him with no problem. Since him nothing. Has anyone heard of this happening to anyone else. I feel like im cursed or something. I will never be able to have anymore kids because im never getting back with him...any feedback will help.