Feel guilty missing the birthday party

I put a date on the group calendar for me, my husband, my mom, sister and her family(husband and 1 year old) to go to my grandmas beach house. I forgot to notify my husband so when his sister sent us the evite for her son’s 1st birthday I realized it was the same weekend.

Without getting into a ton of details as it’s a story in itself, we don’t particularly get along with my SIL and her husband. Everything was fine until something happened over Christmas and now sister’s husband won’t speak to us. My husband also stopped putting up with his sisters BS. They live at my MILs and that’s where we stay when we visit(we’re 4 hours away).

We witnessed something few weeks ago that my husband didn’t agree with and on the way home he was venting and said “maybe we’ll just be busy the weekend of the birthday, I don’t care to see their new house or be around that” I brought up how we could go to my grandmas beach that weekend because my mom and sister were planning to be there. He tells me I can go, nephew won’t remember I wasn’t at his first birthday. It’s an extended family birthday. Basically saying he’s going to the birthday but I can skip.

I briefly mentioned it to my mom and she said that we could move the date because my sister would be there the weekend before. Turns out my sister isn’t going the weekend before and my mom essentially told me yesterday that they were still going and “you put the date on the calendar”

What do I do? It’s my sister kids first time at the beach house we went to as kids and I’d like to go but I feel guilty missing the birthday party.