My boyfriend hangs around bad people.

I’ve been with my boyfriend for almost 2 months and we hang around each other during school, but lately he’s been wanting me to hang out with his friends, and not mine. I immediately said no to this because they are terrifying to me. As a young person with anxiety and who has been harassed by these people multiple times, I do not feel comfortable being around people who smoke and do drugs. I feel like they’re going to hurt me, like I’m getting involved with the wrong kind of people. Almost all of his friends are like that and I get so anxious thinking about what I’m getting myself into. I don’t want to break up with him, but I don’t want to feel like my safety is compromised. What do I do?