Husband smokes weed

My husband smokes weed at night and since I have gotten pregnant, he smokes weed outside. He only usually takes a hit from a pipe and then comes inside after putting his thumb on the weed to put it out. When he comes inside though, it smells like weed in the house for a few minutes. I don’t really know what else he should do, but should I be worried? It’s really frustrating cause of all responsibility that falls on me to ensure we have a healthy baby. Any thoughts?