infertility?

It is not a big deal we are still young, my boyfriend is about to turn 24 and I'm 21, we have been together three years and in the beginning of our relationship a condom broke, I have a 5 year old and the doctors told me I'd be extra fertile so I assumed I was pregnant but I wasn't. We haven't used condoms since then and we have sex at least a decent amount of times. Not trying to go to the Dr and have tests done but just wonder why it hasn't just happened yet.