Apparently I'm lame for not smoking

So me and my boyfriend just got finished having sex and we both had a orgasm after we were finished my boyfriend suggested smoking weed so we can have some wild crazy sex and I say no cause I'm just not into smoking. I've smoked in the past with him but that was maybe a year ago and I've also had edibles. So since I said no he said I'm lame and never want to have fun and I told him we'll go date someone that smokes then and he said fuck you. Am I wrong for saying no? He doesn't like to drink and I never pressure him into drinking I just don't understand why he can't just take no for an answer.