SO vs Family

My SO got into it with my father and grandmother, in which he was really disrespected, and in my opinion they went in way harder than necessary. It happened maybe 6 months ago, and ever since then he's cordial with them but doesn't wanna go out with them, doesn't wanna hang out for long periods of time. None of that, and if that's what he wants to do then I'm okay with it. They said the most rudest things to him, never apologized and accepted his apology even though he didn't do anything wrong to begin with. However my issue is now, that even though I'm on his side and I've expressed that they shouldn't have spoken to him like that, he gets upset with me when I go out with them or hang out over their house. And this is where I'm torn because HELLO this is my family, but HELLO they're so rude and never apologized. What should I do or what do you think?