Feeling left out in my own family... am I wrong for this?

I get invited to most family get togethers, and they’ll act upset and offended if I don’t attend, but I usually always go and when I go I feel left out. My SIL is usually who all the women in my family want to talk to most. Probably because she has a toddler and I don’t have kids. But they even talk to her more when the topic isn’t about her kid. Like I’m usually just sitting around not getting to say anything because I can’t get a word in edgewise, I try though but sometimes I’m literally ignored. This is how it’s always been but I’m getting really sick of it now. The men in my family usually watch sports and I don’t care for that. My nephew loves seeing me and I do spend time with him but sometimes I feel like a babysitter instead of included in my family. My boyfriend who I live with won’t attend my family get togethers, hasn’t in like 2 years, because either he’s working or he just makes an excuse. He has nothing against my family (so he says) and they know that, he’s just a hermit. Whatever honestly.

Thanksgiving is usually at my parents house but apparently it’s at my sil’s house this year. Nobody told me. The women in my family were just talking about it, obviously already knowing about it, while I just sat there like what? And my sil later just turns to me and she’s like you’re coming right? You know what. I don’t think I’m going to. I talked to my boyfriend about it and I told him I’d rather go out to a restaurant with me and him and maybe his cousin and his girlfriend for thanksgiving (the only family he has in this state). I have never not attended my family’s thanksgiving but I really don’t want to anymore. Am I wrong for this decision?