Boyfriend wants visit daughter in town with ex

ivory

So we’ve been together almost a year and even live together. His daughter lives 5 hours away. I have 3 daughters who live with us and so he says he wants to spend alone time with his daughter with no distractions. She’s 4. I love her to death and she loves coming over. Thing is his ex still loves him and has texted him called him even cried to his mom she misses him. I don’t want to be selfish but don’t want him alone out there with his ex. He cares for his daughters mom to not see her hurting but she is capable of anything to get him back. And her knowing he’s out there without me who knows. I know it’s not about me and need to trust him but I’m afraid she might want try to make him feel guilty and start her crying and he’ll spend time with her. I want to be a woman and not a immature little girl so agreed with him to go. I just need help on what to do to keep

Any negative thoughts out of my head. Should I be upset for him not bringing her ??? Should I just not say anything and let him go alone?? I just don’t want him to start doing this every weekend and start keeping his daughter away from us. I want her part of our lives.