Male Fertility and Marijuana

Hi, I am looking for some advice on this. My SO and I are trying to convince. This is our 3rd month trying and it hasn’t happened for us. I know it might be to soon to start to worry but, my SO smokes weed a lot. He smokes 3-4 times a day. And I am very worried that we might have a problem conceiving because of that. Has anyone else had this problem? Was it difficult to conceive with someone that uses marijuana a lot?