What do I do

So my partner and I just had a baby and had a postnatal diagnosis of Down syndrome. I have a 3 year old but he’s not the dad.

We don’t currently live together and he comes and visits and stays in weekends... he finishes at 2pm and comes around at 7pm/6pm and makes sure he leaves at 8pm/9pm. He saw me in hysterics the other day, and he asked to go to his mates the next day... he then says he has had no affection for 3 months (my baby is 4 weeks old) and tries kissing me and feeling me and I say no and he keeps going like I have to say “no” like 10x... he does nothing when he’s here, I need to tell him to do everything like he has no initiative. I don’t want to be with him but I’m so scared to be alone (even tho I partly feel alone) he’s also not coming to the baby’s appointments (because of her diagnosis) because his work is more important.!

I feel like I’m dating a 15 year old.