Smoking pot

Soooo this guy I like smokes pot. I never smoked and I don’t have any plans to. He never pressured me to or anything but I feel as if he will get bored of me because I don’t smoke. All of his friends do and my older sister is friends with him and she smokes with him sometimes. So he’s having a get together with a few people with drinking and drugs and I’m not about that life so I don’t want to go or be associated with that. I have extremely bad social anxiety and I feel like my sister thinks I’m boring because I don’t want to smoke or be bad. I just don’t want to!! Why would I want to go to this thing if I’m going to be standing there doing nothing with a whole bunch of people I don’t like/don’t know and get anxiety from?? Why would I put myself through that. My sister has anxiety but I feel like she doesn’t have that bad of social anxiety because she’s still pretty social. I don’t know what to do but I don’t want anyone to be mad at me and I don’t want to smoke because I just don’t want to! It’s really not that deep but I don’t know. I feel uncomfortable hanging around when I’m doing nothing. I’m just not a bad kid. I’m an anxious child I always have been and I always will be. I’m always told to loosen up by my sister but I have social anxiety and I don’t want to be around that when I know it’s going to fuck me up (anxiety attack wise) for days!! I just don’t know what to do. Please help