Am I insecure and denying it?

I recently started dated my boyfriend but he has a child with someone else. We live three hours away from each other but he lives in the same city as his child and his child's mother. I have trust issues with him because to me, she's just another female that he always has to talk to, a female who gave him his first child, so I feel they have a special connection. But they haven't been together for almost a year now, and he gets annoyed when I keep bringing their relationship up. He claims I have nothing to worry about because that's the past and he wouldn't have started a whole new relationship with me, if he still wanted to things out with her. But what women doesn't want her original family to be together. Should I just relax or should I be worried