A bit controversial but pregnancy related

Ok, so I'm almost 39 and my fiancé just turned 44. We want a baby as soon as we get married. We are hoping to get pregnant on our honeymoon in just three weeks. I am trying to quit smoking to prepare for this bi had my Nexplanon removed in preparation. My fiancé has smoked pot for nearly 30 years. He says he's trying to quit but hasn't yet. Pot diminishes sperm, misforms it, and often makes a man sterile. So that, on top of my PCOS, is going to make getting pregnant very difficult. I've lost a lot of weight to help the PCOS. My question though, is if by some miracle we do get pregnant while he's smoking pot, does that sperm have a chance of causing the baby to be malformed or sick or even come out addicted to pot already? Or is the addiction thing only when the mother is doing drugs while pregnant? If he manages to have just one sperm that makes it to my egg and fertilizes, will I need to fear for my baby? I don't like him smoking it but he's so addicted we get into WWIII when he tries to quit. I'm trying to determine the lesser of two evils? Advice much appreciated.