Husband smokes a lot of pot

We've only been trying to conceive number 2 for a few months now. My husband smokes a few times a day. He doesn't get "high" he just smokes. He argues with me that this has nothing to do with me not getting pregnant. He thinks because we conceived the first time after a few months that nothing is wrong with him. That it has nothing to do with smoking. I'm just feeling so down after seeing a negative this morning. I feel like he's not supportive. He says he is and he does want this. His response to the negative was "well we just need to have more sex, that's not a bad things" he's always thinking like a guy and I can't fault him for that. He said if we don't get pregnant for 6 months then he would cut down on smoking. its just so frustrating. I never went on any sort of birth control after having my daughter. So there's nothing my body needs to flush out or anything. I guess I just expected for it to happen sooner