His Friends

I get really anxious when he asks me if I wanna go and hang out with his mates. I've had some bad luck with friendships the past year or 2 as I moved away and everyone just stopped speaking to me. I find it hard to make new friendships and I'm scared his going to get mad at me if I continue to make up excuses as to why I can't or don't want to hang out with his friends and their girlfriends. Any advice on how to feel better and confident in hanging out with them?? I might sound silly! Thanks xx