Is anyone else TTC with Varicoceles?

My husband was told a few years ago he may be infertal because of his enlarged veins in his left ball sack... the doctor told him it was called varicoceles. And we've been having unprotected sex for over 3 years, just using the pull out method, and we've never had any unplanned pregnancies.... and that's worrisome to me. Now that we are thinking about trying to conceive, I'm worried it will take years... and I've always dreamed about a big family... he also doesn't want to get tested until we've actively been trying for a while... So I just needed to seek advise and stuff...