Legal help please need advice

So I have 2 kids (1&3) from my previous relationship. When their father and I were together we had to put a custody agreement in effect cause I was joining the military and my recruiter told me that he needed proof that someone would be watching my children. We decided that we would share partial custody. I've taken care of the children everyday since they were born premature even when he and I lived together I still did everything while he sat back. My son was born 28 wks and I was in the hospital with him every day while his father wasn't. The same for our daughter who was also born early. They have been in and out the hospital since birth and I have always been by their sides never missing a day. I have handled every appointment and every therapy. When my son was told he would never walk again I ensured that he would, while their father set back and did nothing but smoke and work. I finally couldn't take it anymore and left him I decided to stay in the house cause it was familiar to my children and I didn't want to just take them from their home. He began to become more and more controlling so I couldn't stay in our home anymore. I took my kids and left. It was hard for me to get him to watch our children it was always an excuse as to why he couldn't do it or he just wouldn't show up. Times when he did he would bring my kids back in poor condition. My 3yr son has cerebral palsy and it was difficult to get him to potty train and every time he would go with his father he would come back in diapers. My 1yr daughter came back with a dirt rash that ultimately left her skin scarred. I felt completely helpless cause by law I couldn't keep my children away from him. All the court kept saying was " you can file for a filing date to modify custody.". The brought my daughter back without 3 missing teeth 2 of them were just coming in. Now my baby girl has front teeth because he was negligent. The last straw was when he had 10 people jump my current bf, threaten my brother, and my children. I fled to a different state with my children and bf. I had to put a restraining on him and he never showed up to court. He told me he wants nothing to do with the kids and now he's threatening to report me for kidnapping and asking to see the kids. In scared that if I allow him to get the kids for a little that he will neglect them like always and that he will keep them from me. Please can anyone help me I really need the help I don't know what else to do I'm pregnant now with my my boyfriend baby and I can't focus on this baby as much as I want to cause of my ex. Sorry it's so long I wanted to fill anyone in as much as possible. I really need to talk to someone