Is it wrong not to sit with “family” at a sporting event?

So I had a falling out with my husband’s parents (who greatly influence his 11-year-old sister who hates me now). I still go to his little brother’s sporting events because he and I are on great terms.

I made it through the last several games of the football season sitting with them, but powerlifting is coming up, and that’s a different situation.

In football, you can watch the game the whole time and not really have to talk. In powerlifting, you sit there for several hours to only watch your person lift once every hour and a half or so. So basically lots of down time where I know his family will only talk to him and/or make underhanded remarks about me or things I do. I just don’t want to go through that every Saturday! I want so much to support his brother and be there (especially since i couldn’t make it last year because i was super sick for several weeks). But my husband thinks we always need to sit with his family so his grandparents don’t question what’s going on. I’m just looking for how other people think about this situation.