My boyfriend said he would break up with me if I smoked weed

My boyfriend from the very beginning of our relationship told me he hated weed and couldn’t date a smoker. I used to smoke and he was actually the reason I quit. We’ve been dating for almost two years now and lately I’ve really been wanting to smoke. I actually did hit my friends dap pen a few weeks ago, didn’t even get high and ended up telling my boyfriend the day after because I felt so guilty. He was understanding, but told me to never do it again. I expressed to him that night that I still want to smoke, it calms my nerves and honestly it’s fun to me. It makes me feel like he doesn’t accept every part of me since he doesn’t allow me to smoke. He also is pretty controlling and for the longest time he wouldn’t even allow me to wear earrings or hangout with certain people. I love him a lot and don’t want to do something that is going to cause us to break up but I find it unfair that a bunch of his friends and even his brother smoke but he only has a problem with me. Should I talk to him about it again or am I in the wrong for feeling this way. I’m torn and feeling hopeless constantly because I feel like I’m the problem in this situation