Advice please!

Hi! me and my boyfriend have been together just over a year, he's the absolute best like I've never loved anyone as much him, literally so obsessed with him! we have a 17 week old baby. everything is going great although we don't live together (yet) as he looks after his daughter (she has autism) in the week, he comes most weekends. the problem is he doesn't say no to his ex, as in when she asks to keep their daughter during weekends he says yes. No thoughts about me and the baby. he does everything she says. he slags her off to me, called her the worst names possible yet if he isn't messaging her (about their daughter mostly) he's mentioning her name often! I don't know if I'm being silly and over reacting about this? I've told him how I feel his response was that his daughter's mum is next to useless so hes left to care for her by himself. Yet here I am with two children (including our baby) all by myself. Can somebody please give some advice on what I can possibly do? It's really getting me down lately 😔