Exes new girlfriends

I've recently started dating a man who is the polar opposite of my ex. He is absolutely amazing. Kind, caring, heavily family oriented, close with his mom and dad and siblings, and has a 5 year old son that he has raised alone. (his ex wife ran out on them when their baby was 3 months old) He's a firefighter and also helps work with disabled children. He's made it very apparent that he's marriage minded and bends over backwards for me. He leaves grocery money for me and even tries to help me clean. My family loves him and my best friend adores him...which is huge considering she hates pretty much everyone 😂 

I was very hesitant about him meeting my 3 month old but the first time he met her was at my parents house and last night we went to his house to watch Disney movies and, of course, my ex calls. (For the first time in over a week) Instead of lying, I was upfront and he freaked out. Calling me a slut and a whore and suddenly was quite concerned about our daughter. He has put no money in for diapers, formula...nothing. He owns nothing to take care of her and was demanding that he be allowed to watch her while I supple everything. He told me that his new girlfriends are "dumbasses" (his words) saying they lack common sense and are just unintelligent but expects me to leave our daughter with him for extended amounts of time while they all hangout. I He is now saying he is debating on if he even wants to be in her life because a) he doesn't want ties to me and b) he wasn't ready for children. 

I flat out told him that his girlfriends won't be meeting our daughter for a very long time for the fact that a) none of them have experience with small children, b) he has a history of dating for fun with no commitment (I was his first serious relationship) and c) He has a history of having his home broken into due to him opening his mouth and being to flashy with his shit. Last year we were held at gunpoint while men ransacked his home and I was raped. 

At what point does one draw the line? I really do want her to have a relationship with him and I wouldn't have a problem with our daughter meeting a girlfriend of his IF they were serious. I have custody of my daughter full time and his name isn't anywhere on any of her paperwork. Am I totally wrong for not being comfortable with my daughter being left alone with him and his new girlfriends?