Questions about fertility TTC

I’m not looking for judgment please just need to clear my mind because I can’t stop thinking about bad things ! I’m 21 and my husband is 29 he has a daughter who is 13 (he had her when he was 15) and that’s his only child (so far lol) so I know that he is good and I had a pregnancy when I was 18 so I know we both aren’t infertile but he is a recovered drug addict. He did crystal meth and heroine for 13 years and is 1 1/2 clean. I have never done drugs other than marijuana and we both quit smoking it to TTC we both smoked cigarettes prior as well and I just keep hearing that smoking can lead to infertility. Is this super accurate ? I know someone who just had a baby who smoked but it makes me nervous when I hear about these infertility problems due to drugs or smoking. Keep in mind he doesn’t do drugs anymore and we both don’t smoke anymore but it just makes me nervous! Help me clear my mind pls :(