Baby daddy drama

So I have 2 baby daddies, my son is having his birthday party and wants to invite his dad I said that was ok, but my other baby daddy doesn’t want our daughter going because he doesn’t want her around the family of the other baby daddy, I told him it’s for her brothers birthday and that he should understand but he was all like “imma have a second baby momma and do the same shit you’re doing so you can see how it feels” anyways that escalated pretty quick and I was shocked and at the end of the day I don’t wanna get him mad but we are not together and I feel like I can do whatever I want. Idk what to do