Broke up with boyfriend because of his extended family

So I broke up with my boyfriend because I could not deal with him and his extended family he has with his bm. I feel like he makes not only his kid a priority ( as he definitely should)but makes his kids mom a priority of his as well. They still celebrate each other bdays, spend holidays together, he still sleeps over her house ( if his son who's 4 asks him to stay, he says🤔or he falls asleep accidentally 🤔) and go on vacations as a family and it all makes me super uncomfortable. When I talk to him about it he's always like" this is my family and I'm going to be there for my family." And, I get that to an extent but I thought when you got into new serious relationships the two of you worked on building and/or creating your own family. Not saying he can't see or spend some time with his bm, as his "family" as he says. But I just feel that he's putting her first because of the child they have together and doing a little much. He tells me he loves me and once I meet her I will understand that their just really good friends. But, even then I told him that's not going to change how I feel .I need to feel like I'm priority in his life, it's been 5 years and I m just tired of feeling that way so I left. I don't know what else to do.