Infertility?

I have ALWAYS had the biggest fear of not being able to have children.. although I have never had any reason to think this (periods have always been on time, etc.) it still weighs on my mind.

I have been with my boyfriend for a little over two years, and for the past year and a half we haven't used any protection at all. We're not actively TTC but whatever happens, happens. I don't want to go too deep into detail, but it is almost like we ARE actually TTC because we have sex a lot around the time I'm ovulating, and as I said before, never use any form of BC, not withdrawal or anything.

I have mentioned this to a few of my friends, and the first thing they all say is "Maybe you can't have kids." and that scares the crap out of me..

So I guess my question is, should I be concerned that he or I may not be able to have children? Or am I totally overreacting?