My bf is so mad at me 😬

Sooo a few months ago we got into an argument and my boyfriend said some REALLY disrespectful things about my brother who is gay (remarks relating to his sexuality). Fast forward to today, my family is going to Atlanta for my dads birthday and we are all staying at my brother’s house. Well my brother doesn’t care for my bf b/c he cheated on me through about 75% of our relationship including the entire time I was pregnant and he doesn’t like the way he’s treated me at times in the past. I didn’t invite my bf to go to ATL and also never mentioned him coming. He asked about it yesterday and I told him I didn’t invite him because I know they don’t care for each other and I still feel offended about the things he said, so I didn’t feel comfortable bringing him to stay at my brothers home with the entire family. I probably should have mentioned it when I told him about the trip a couple wks ago but I just neglected/chose not to do so and now he is angry. I guess I feel bad for making him uncomfortable in having to ask why he wasn’t invited but I also feel like me telling him was an opportunity for him to apologize or make things right about the comments he said about my brother. Ugh I wish I didn’t care but I hate when he’s upset with me 😩