Boyfriend does sketchy things, am I crazy?

So he has this friend group that I go and hang out with occasionally as well, there’s 3 others besides us. The other 2 girls in the group—one he used to like and still fawns over, like he follows her around and ignores me for her & stuff. The other he slept with right before we dated. Last night he hung out with them “to drink” & ended up staying the night with just the 2 girls even though he had opportunities to go home and knew (as we’ve talked about this many times) I would be upset if he stayed over. I don’t think anything would happen but he knows I have severe anxiety, one of our former friends that’s no longer in the group tried to start a rumor that they had a threesome. It was proven false but still messes with my anxiety. I just don’t think he should be spending the night WHILE DRINKING, with someone he used to like and someone he slept with, while he’s in a relationship. Like, I’m trying to tell him that it’s not normal for a dude in a relationship to do that stuff. Especially since he KNEW that I’d be upset that he did that and did it anyways. He’s abandoned me while I drove to see him before. Like we were watching a movie & they were like “on our way be ready” & instead of telling them he forgot about the plans or even telling me that he had made other plans on top of ours, he got changed and left me at his house to go practice driving with them (he’s 24 and still no license) like I’m glad he went to practice but he already did the night before as well with them and knew that we had plans to hang out the next day, which he was quick to accept their offer without even thinking about how I’d feel. I’m glad that he has friends but I feel like I’ll never measure up to them :/ am I crazy for being upset that he spent the night with them? I also told him last that I felt like he didn’t care sometimes and he left it on read and didn’t say anything for 2 hours :/ . Like how do you read something like that and happily go back to drinking and playing a game :/ we’ll be at 2 years in September but I keep having doubts :(