Idk what to do

I broke up with my ex about 3 weeks ago now and I started talking to someone else already. I’ve been over my ex for a good year now, he’s really mentally/verbally abusive and I’m just sick of it. We have an 8 month old daughter together and we are renting a house. The lease isn’t up for almost a year and my ex is really upset that I started talking to someone else and said that I should have waited until the lease is up. Im not on the lease so I can leave whenever I want but he can’t afford it on his own so I feel bad. Am I in the wrong ? :( I won’t be introducing him to my daughter for a really long time. But my ex said that he’s going to take my daughter away from me now because he doesn’t want all these guys in and out of her life. I think he’s just being manipulative and controlling. But I still feel a little guilty :/ I just want to be happy :( I don’t have a proper job right now (I babysit) so he said that he will get full custody of her. Should I just end things with this guy and wait until the lease is up? :/