Smoking the weed

So we have only been trying for a couple months. But I asked my husband to lay off smoking weed. He didn't. I feel as though he is choosing the drugs over us having a baby. Really has put a rift between us. I don't smoke pot and I am not a fan of it it to begin with and he promised me that he would try and quit. Does anyone else have problems with this. I heard that pot slows sperm down. I am worried that it will make it harder for us to conceive.