Hi guys I need some opinions.

So yesterday my parents had asked me and my boyfriend if we wanted to go out and have dinner with them on Thursday so I asked my boyfriend and he said sure. Mind you we were under the assumption that it was only going to be the 4 of us. Anyways, I come home today from work and my Dad told me that he had invited my brother. A little backstory: my older and much bigger brother has put his hands on me numerous times almost to the point of calling the cops, he calls multitude of names including b***h and c**t and thinks its okay, he's thrown things at me etc... and just to sum it me and my brother have a lot of issues and I dont like being around him and neither does my boyfriend because of how he treats me and the things he says to my boyfriend (he calls my boyfriend a thug and says he looks "suspicious" whatever the hell that means) anyways so I told my parents that we don't want to go because of my brother and now I'm supposedly the asshole because im being "rude" by not wanting to be around my brother etc... even though I've told them the reasons why.. They always seem to have my brothers side.. What do you guys think??? Am I in the wrong for not wanting to go now? Or should I just toughen it up and go ???