New boyfriend with kids

I recently started dating someone new. It’s a long distance relationship but we do get to see eachother every other week or so. The most quality time we get to spend together is over FaceTime. One thing I complain about is that. Because when we finally do get time together. His babymomma decides he needs to take their daughter. The first weekend we spent together I went to visit him. &the day before he had gotten his daughter unexpectedly & that was fine. She didn’t love me but she didn’t hate me either. Then the next weekend we got together. He came to visit me I had to work so we just spent time in the hotel after work cause we were tired. But then his bm caught wind he was in town &made him go spend time with the baby. Okay cool I wasn’t all that upset. I was just upset at the fact that I was supposed to get out of work early so we could actually go do something. &he ended up picking me up a couple hours later & we drove back down to take him home &then we spent 2hrs together. Before I left to come back home. Now we’ve been planning to see eachother since last week. Last night he told me he has to get his daughter before he comes &gets me. &we had planned to stay with his cousin the whole time since he has an extra room. But now we can’t do any of what we were planning &im a little upset. Cause I’m the 3months we been together we really haven’t done much of anything.

Am I wrong for being upset? Is there a better way to handle all this?