Should I be worried im infertile

I'm 24 was with my ex 3 1/2 years never used condoms he would cum in me every time we weren't really trying but did want a baby. We would have sex about 5-6 times a week. 
Than my other ex was with him 2 years never used condoms & honestly not sure if he would even cum in me we rarely had sex tho maybe 1x a month. I'm just really scared something is wrong with me I have regular 28 day cycles & my whole family pops kids out like nothing. 
What could be the reasons it has never happend Iv never even had a pregnancy scare. 
I mean I'm thankful now I didn't get pregnant with both my exs but still worries me that I cant now or in the future with my current bf of 1 year ( he pulls out ) we want kids in the near future but it's always on the back of my mind I won't be able to. Any stories like mine that were successful am I just acting crazy. 
& please don't tell me to ask a doctor or that I have to be actively trying for 2 year I know all that. Just need some advice or encouragement.