Other ladies views

We are a blended family . So things tend to get difficult . My current situation is cancelling trips or events because my so ex says there child can't go with us . Its starting to cause a problem because there's other kids to and there fun gets cancelled because of that . I don't even know how to approach the situation. It's like our life revovles around her . I get that's there child but I don't want to put our kids through that there whole life . What am I supposed to tell them "oh sorry kids but we can't go because your brothers mom won't let him come "