Do you understand me or am I overreacting?

Okay so my boyfriend hangs out with him brother all the time 1-2 times a week and sometimes even 3 times. I don’t mind it but when ever I come over to my boyfriends house, there’s at least once we go to his brother house, I wouldn’t mind but his brother is such a negative person. He has short temper and it’s very uncomfortable. He gets annoyed of his daughter and is always telling her to shut up or to go to her room. Mind you shes only 6. She will play in the living room with her toys and if she’s making a mess she will get yelled at by him. And he also yells at their dog. It’s very uncomfortable and negative environment. And I honestly feel bad for the daughter. Sometimes theirs even tension between my boyfriends brother and his “girlfriend”. Their not actually together but live together for the daughter. They just act like they are together around family. Family went over for the super bowl and we were all just hanging out and having fun and then we hear my boyfriends brother yelling at his “girlfriend” and she’s yelling at him back and she ended leaving to the room and making a huge scene. We all decided to leave 30 minutes after because everyone felt awkward. And I had to convince my boyfriend to leave by telling him I need to leave early because of school. Now the thing is, my boyfriend wants to go to his brothers today to go watch some ufc fights. I really don’t want to go and I have voiced that to my boyfriend and he just got mad. I don’t think he understands why I don’t want to go but I have told him multiple times that I don’t feel comfortable being there and why I don’t feel comfortable. Since his brother has a short temper, my boyfriend tends to not tell certain things because he knows his brother will get very mad. I also don’t like my boyfriend going over because I feel like his brother influences him and they tend to exclude themselves from family sometimes and be antisocial. My boyfriend grew up very sheltered and I want him to socialize more and not be so closed off but him always being with his brother. They always tend to do that.

Am I just overreacting or would you guys feel uncomfortable as well? My boyfriend is upset with me because I didn’t want to go and decided to go to sleep.