Issues with baby father please help

My partner and I have been together off and on for 4 years. This may sound dumb as it has yet to happen but the things he says imprints in my head regardless.. I have a child who is 7 with someone else, he never really bonds with her and that has always been a “problem”. So my partner makes comments to me that if things don’t work out between us he wants to take our child, and his reason to that is “you have your other child so you won’t feel alone” I was taken back by that dumb comment, I told him no that it doesn’t work that way. Just because I have another child doesn’t mean anything. This child will be just as much mine as his and these comments are very aggravating to me. He always makes comments that since I have a child already he should have the right to do more with this child than myself. I’m just looking for advice on this situation because no nothing has happened yet but IF it did I’m almost certain he would have this mentality and idk how to handle it.