Am I wrong ?

I've been with my boyfriend (now ex we broke up today) for 5 years. We have 2 children together. Our son is 3 almost 4, and our daughter is 4 almost 5 months. We argued today because I don't feel like it's okay that his ex is still in the picture. She will be attending our friends wedding, and his sisters baby shower. She still speaks to his mom, his sisters and his brother. Why can't I feel like she is a threat to our relationship? Am I wrong for feeling like she shouldn't be around anymore? His mom still speaks to her, and granted we can't control who she speaks to, but he should tell her that his ex being around is hindering our relationship. And his mother should understand & put her sons happiness before anything. He thinks I'm completely in the wrong though, so we broke up today... 🙁