Social anxiety or just shy?

Sorry this is a long post so bear with me. I’ve always been quiet even as a kid, but I think as a kid it was easier for me to talk to other kids rather than now as I am in my 20s. Even middle school and high school I was shy but I still talked more than I do now. I live with my bf and he’s very outgoing. He loves to go out and do things with his family and friends. I’ve met a lot of his family members and his friends and they’re all nice. Today he wanted me to go with him to go to his friends house and hangout but I couldn’t do it. I just got really anxious, felt like I couldn’t breathe, and got nervous. Even when I do go to events with him, I basically stay quiet. I worry about what other ppl think of me, I worry about people staring at me, i worry about embarrassing myself too. My bf and I get into it because he thinks it’s rude to not go and I see what he means but it’s really hard for me. Any advice?