Relationship problem pleaseee read, going insane

This is long but I just need a second opinion because I am just so confused and hurt. My boyfriend and I have been together for 3 years. To cut to the chase, his twin brother who is going into the marines in a few months hates me. My family thinks it's because he's jealous but I don't know. This is just something you'll have to take my word for but I have done nothing to him. To put it into perspective, the rest of his family loves me. For the past year or so my boyfriends brother has tried to convince him to cheat on me, break up with me, and relentlessly talks bad about me to my boyfriend and his family. This really hurt me because I am not confrontational and I am a very nice person. Despite not doing anything to him, I have still apologized many times for whatever reason his brother hated me that week and tried to make up with him. Well my boyfriends sister is visiting from out of state with her new boyfriend and wanted to organize a family dinner, which included my boyfriend, me, his brother, his sister from out of state and her boyfriend, and his sister about an hour away and her husband and son, and his parents. His brother decided he won't go if I'll go. My question is do you think my boyfriend should've still gone? It's not like i was intruding, all the other significant others were there. So should my boyfriend have supported me instead of condoning this behavior because he knows full well I have done nothing to his brother? It's not like there was a final family get together and I was like "no, I don't want to go you should stay with me" it was everyone was invited and it was his brothers decision to not go if I go, so if it was really centered around family values then it shouldn't have mattered to his brother if I was there or not.