smoking tearing us apart:(

Hey guys, I know this might be a touchy subject but I just wanted some insight. Before I say anything I want to say that I AM NOT against weed. But I've had friends and have known people who used it the wrong ways and have me some bad experiences with it. I knew my fiancé smoked weed and I didn't have an issue with it, I smoked with him occasionally as well. But one day he started smoking weed from this new "source he found" and it made him weird, like he didn't care about me and we couldn't ever have a conversation because he showed no interest in like, anything. He was high all the time, and I told him we need to set some times when I can get him sober sometimes. He didn't care about this either and didn't want to give me any time I could talk to him normally while he's sober. This went on for a few months and I didn't know what else to do so I broke up with him. Our breakup didn't last long but it made him realize I was serious and I needed some time with him when he's in his own mind for us to work. Since then I think we both have grown a lot while being together, jes a wonderful man and partner and we are happy. he still smokes but I stopped smoking weed since our breakup. A few months ago he expressed to me he wants to start smoking weed with his buddies again. I don't know why but for some reason this made me really really uncomfortable. I knew that it's okay for him to smoke but for some reason I got so uncomfortable and anxious and it made me remember our breakup when I couldn't ever get a intelligent conversation with him and I felt scared. I knew that my fear was unreasonable so whenever he smokes or is high I leave the room or I just leave and come back when he's sober. He's told me that because of that he hides weed and doesn't talk about it around me. He says "Your just one of those simple minded people who doesn't approve of weed" but that's not true! I use to smoke occasionally too! I've always tried to be outgoing and try new things and the fact that my uncomfortableness with weed is because of our breakup and he doesn't get that makes me really sad, like he doesn't get it at all. I feel like this is making us not as close and I really hate it and I don't know what to do, I feel hurt that he doesn't get why I'm uncomfortable and it doesn't seem like he has my desire to understand either. What do I do?