am I wrong for hating my boyfriend smoking weed?

okay so me and my boyfriend have been together nearly 4 years, we both smoked (although I didn't smoke weed) but I got pregnant and quit (later than I should but I still did it) now he agreed to quit with me, and that he wanted to have quit before our daughter arrived. completely understand how hard it would be but it's like he doesn't even try. every time I say something about it, I get told I'm not his mum and can't tell him what to do. I'm furious because all his friends smoke so when he sees them he smokes more than usual and it makes it look like I don't want him to have friends, which is completely not true. I hate it because I won't allow him near our daughter for at least half an hour after smoking which of course means I have to do everything! our daughter is now 16 weeks old and he has never had her alone because I'm not sure I would feel comfortable. I love him with all my heart and he's amazin. with our girl. it's the smoking, it causes so any petty arguments and I just want it to stop.