Am I being too careful?

My SO smokes weed and I told him that I'm not comfortable with him smoking while we are TTC. Last month right before my ovulation week (which he knew was coming) he smoked. When my peak day came he tried to initiate sex but I told him no. We talked and I cried- we had been waiting until January to start trying because of my work schedule and I guess I was emotional about losing out on our first chance. He felt really bad and said he "didn't know I was that serious about it." He said he wouldn't smoke and would "clean out his system" before our February ovulation week. Well here we are and he didn't do what he said. He last smoked a week ago, and he hasn't taken care of his own "business". He wants to BD, and told me he wants me to have a baby, and that I'm being paranoid. Am I? We're married and in our early 30's- I just don't want any cards stacked against us you know?