Should we be worried about possible infertility?

My SO and I have decided to start talking about the possibility of TTC. The other night, he told me there was a chance that he couldn’t have kids (His brother can’t have kids and his mom told him there was a possibility that he can’t either) and that if it was going to be harder to get pregnant, he’d want to start trying immediately. I don’t want him to have to get tested now, since we aren’t even TTC yet.

The only sort of birth control we’ve ever used is pulling out, and even then, he’s cum in me a lot of times, even when I’ve been fertile. Is this something we should take seriously and worry about, or is this normal?