Opposites

So this guy im seeing is basically the complete opposite of me. We have fun together but he has one quirk i do not like at all. He smokes weed. I told him he can do it when im not around and dont be high when he is with me. I know weed isnt bad line other drugs I just grew up with a drug addict and i am completely over being anywhere near potheads or any other types out there. He agreed and said he understands. Well now he wants me to spend the night and I didnt have a problem with that but then he said he wanted ti smoke before bed so i told him im leaving. Hes like why im not going to talk to you itll just help me sleep. I dont get that as all I can rub his head till he falls asleep or something to help him sleep. I just felt like he chose pot over me because i told him if he wants to that fine im leaving and he let me leave. I feel like im just over reacting and projecting my past onto him. Am i being silly or am i right to feel like i wasnt picked over that?