Dealing with marihuana

My boyfriend and I have been together for a year now. It's been almost perfect, we complement each other and have fun together. The thing is, he was in a band before and they used to drink and smoke a lot. He and his cousins and some family smoke marihuana too sometimes and honestly, I've never been that kind of girl. I didn't really mind because I love him so much, I even tried it twice and to be honest, it wasn't like the best thing in the world. I told him I didn't really like it because I was depressed after doing it and even him too. He acted different and made me feel like I didn't know him. It's become a thing between us because he knows how much it bothers me when he do it. I also don't like to be the kind of girl telling him what to do or not. I went trough it before with another guy and it was horrible. I don't know what else to do.. Or if it's just me who doesn't want that so should I be the one telling him not to be with me so he can be "free"? I love him so much and j know he is so much more than that. Also I know marihuana is not "heroin" or "cocaine" but I just don't like it. What should I do?