Why does this bother me so much?

I found out that my husband has the occasional drag of weed when he drinks with his work buddies on a Friday. This makes me SO angry but I can't figure out why.

He's a 37 year old man, he's allowed to do whatever he wants but he knows what my stance is on recreational drugs (even weed). When he used to smoke weed ages and ages ago he used to get so aggressive and confrontational and nasty so he stopped. He used to smoke weed all the time when we first met, I'm such a square I've had one drag once when I was like 16.

He tells me no he doesn't do anything but then I hear he does, but that's not even the part that bothers me.

We're TTC and we both have fertility issues, so he's on VERY expensive sperm supplements (prescription) and while I feel then it's pointless if he's going to be smoking, that still isn't the part that bothers me the most about it!

He was with female work colleagues as well, drinking and smoking but I'm not even sure that's the reason why I'm so angry about it.

I really can't figure out why it bothers me so much. Is it because I'm sitting at home, alone on a Friday, waiting for him while he does this? Is it because of the past? Is it actually because he's lied to me about it? Is it because we're supposed to be TTC? Is it all of the above? I can't pinpoint an actual thing that makes me angry about it.