Fertility and Marijuana

I'm a little curious to know if anyone else has had trouble ttc with their significant other that smokes marijuana?? It seems at this point this is the only thing that might be holding us back. 
Yes he has his medical card so it's legal. 
Thanks!!