Advise please, don't know what way to turn..

I have a wonderful nearly 5 year old boy to an ex. His father is very part-time so I have raised him pretty much alone. He's such a kind and loving boy. I then met a lovely man who has two boys. We have moved in together and have baby on the way. Sadly one of his boys really struggles with anxiety and is autistic. Things have started to get really bad and he's becoming violent his other son as a byproduct is very effected. They are broken little boys 😖His dad is stressed out and taking things out on me. They live with us half the week and now we are having them an extra night as their mum can't cope. I could cope knowing there was a little respite and that my boy had time without seeing meltdowns. My partner recently expressed his regrets for selling his house and buying a house with me as it's effected the boys. I'm so hurt and sad and feel and over whelming worry for the safely of my boy and baby. (His boys are not happy with pregnancy either) I feel that our relationship is really effecting his boys. I love my partners dearly and care for his boys welfare as well as mine but with all this stress and signs of regret I wonder if I should gracefully step aside and let them be with their dad (without me) and before children get attached to the new baby. What way do I turn?