Am I wrong?

Destiny • Two beautiful angels and my rainbow girl!
My fiance and I smoked weed we found out we were pregnant and I smoked a few times only taking one or two hits because I was told it doesn't hurt the baby but it helps nausea just as long as it's not in my system when I go to the doctor (normal). So today when pest control came into my apt. I completely freaked because they almost opened the draw my fiance keeps that kind of stuff in. I was able to move everything without them seeing but this being my second pregnancy and my only baby to show for it which I'm still 6wks so still in the freak out stage, it really upset me. Especially since I've talked to him about stopping multiple times and we agreed no more weed after a positive test. I refuse to lose my baby and someone reporting that we have that kind of stuff in our house puts our baby at risk I feel. Do I have a right to be upset or should he be more understanding that I can't go through another loss of one of our children? Btw my first son was lost in a miscarriage.