Varicocele

My husband found out he had varicocele years ago. We weren't trying to concieve when we had our now 2 and half year old son. We started trying 3 months ago and I know it is rather soon but I am starting to worry it may effect our chances. He had the varicocele flare up a few months ago and wondering if they makes a difference. It hasn't bothered him for years .. Like 5 years before this. When I Google it ... It says secondary infertility is even more common than the first time around. So that is causing more worry. Going to try to stay positive. Any one have any advice and or knowledge on this issue?