Need advice please ! In a bad situation

Ok so my boyfriend and I broke up when our daughter was 2 weeks old well actually he walked out. When that happened I didn't know what to do that I called my older daughter's father cause i needed help. At the time I thought it was the right thing to do. When I decided to ho back my daughter was almost 2 months old. Now we all know after pregnancy are mind is not all there. All he wants from me is sex and I keep telling him no! I didnt even tell I wanted to be back together but in his mind we are. I came to him for help but forgetting how crazy he was that that is not possible just being a friend to him. My ex and I agreed on meeting every week so he can see the baby and that's working out great and we are gettinf along so much better and I regret making tue choice that I did and now im scared to leave. My older daughter's father is beyond crazy. See I left him 6 years ago cause he's a control freak and overly obsessed with me. Yes if i leave he'll make sure I dont take her with me. I now dont know what to do I feel trapped! Please someone give me advice on what to do. My daughter thinks she's never gonna see me again. Have in mind that I do have sole custody of her but since I dont have a home right now I was told he could take her from me.