anxiety about conceiving

I'm 26, my husband is 36. we just decided we want to start trying to have a family. during one of our discussions, he made a passing comment that with his age and previous drug use, he may not be fertile. naturally, I was a little anxious and suggested fertility testing and he said he's not worried about it. but now that he's thrown that possibility out there, I am worried about it. when I suggested <a href="https://play.google.com/store/apps/details?id=com.glow.android">tracking ovulation</a> so we can strengthen our chances of getting pregnant, he brushed that aside too, saying that if it happens, it happens. should I be worried at all about this? this would be my first time getting pregnant so please no rude or mocking comments