Starting to hate my brother in law...

My husband and I are very close to his family. His younger brother (28) we used to be very close with. We always hung out, he even came to Vegas with us when we got married. We’ve become distant since my husbands kids moved in with us because we’re so busy but they still come around a couple times a month because we live next door to my husbands parents. His brother is always very harsh with the kids. They’re 12 and 13 and he talks to them like they’re adults. Messes with them, cusses are them, makes fun of them. I ALWAYS step in and tell him to stop, so does my husband. Back in November everyone was hanging out (I was at home sick), I guess my step son made fun of him during a board game and he got SO mad and pushed my son to the ground yelling at him. My son thought it was a joke and laughed about it bc that’s how his uncle is. My husband sent the kids home and started yelling at his brother, obviously pissed. His dad kicked him out and told him to leave it alone and he would handle it. When the kids told me I went next door and after hearing what happened told his parents I wasn’t okay with that and he was no longer to be around the kids until he grew up some and apologized at least. His mom just said “fine”. I texted him and said “Never put your hands on my son again.” Since then, him and his GF rarely speak to us. On thanksgiving we announced we were pregnant. They said nothing. Didn’t get the kids gifts for Christmas, he used to go ALL out for them. Didn’t talk to us at new year. I really just resent them now. He never apologized and it just all blew over. I avoid them any time they’re around. But we’re nice and try hard to talk to them when we’re all together. The kids ADORE him and it just hurts me. We have my step daughters bday party coming up and our baby shower and I’m worried they just won’t show up to anything. Should we have a talk with them? Or maybe his parents? I hate conflict in families and it’s just so tense now.