Why can't I leave? 😔

Lately I feel like my relationship with my husband isn't the best and almost every day I think about breaking it off. Yet I can't bring myself to do it. In the beginning it was great and fantastic. So full of love and happiness and a part of me holds on to those memories in hopes of getting them back. Lately it's been a train wreck. We've been married for a year and together for three. Over the past two years I started noticing a change in him. At first I thought he might have been cheating on me but I found out he wasn't so I left it alone. He started getting meaner and insensitive towards me. He's also VERY UNAFFECTIONATE with me. He doesn't kiss or hug me, he doesn't hold my hand in public like he used to and many times he walks ahead of me like he's embarrassed to be seen with me. He's told me many hurtful things such as he's not attracted to me because of my weight (after giving birth to out daughter) and that's why we don't have sex anymore. That he wanted to be with someone else but he settled for me. Sadly right now I'm not working and I'm a stay at home mom but I've been looking for a job. He told me my thoughts and opinions don't matter because he's the only one working and when I told him when I did have a job they still didn't matter, he told me it's because I didn't make enough. Sadly we're in a bad financial situation and we're moving to my moms house because of the foreclosure of our home. He's told me it's all my fault and in the 3 years we were together I ruined his life and he lost everything because of me. He blames me for everything that goes wrong in his life. And I left many times but he always brings me back. I dont know why but he has this control over me and no matter what he does I always forgive him. I hate myself for it most times because I know what I deserve and I know how I should be treated but I can't leave. Also I feel like a failure if we broke up. We only been married a year and already looking to divorce. I feel like I should try harder. The times I did leave he uses our daughter against me. He says things like if we break up he gets custody of her and if I don't give it up to him he'll take me court. Knowing if he does I won't have the money for lawyers, that I'll be living with someone (my mom) and I won't have a good job so he'll win. I think a part of me is afraid to leave. He's never hit me but told me many times he wants to beat me and if he could he would. He's starting to raise his hand to me as if he's about to strike but doesn't. I dont know if this is abuse because he hasn't hit me but I feel like it is. I also think no one would believe especially his mother. She sees nothing wrong with him. I wish we could go back to when we were madly in love again, I think that's why I stay because he promises that it'll go back to that one day, I just have to be patient. He's even told me we'd go to counseling because he doesn't want it to end but he's not doing his part to go. I'm sorry it's so long but I feel so alone and have no one else to talk to about this. And honestly this has been my third time writing this but I usually delete it. I feel like they're is so much worse in the world for me to be complaining about harsh words. But he makes me feel useless. I dont know what to do any more. 
So finally after ignoring me for a whole week he decided to end things. He said he's moving with his mom while I go live with mine. We argued for a while about who our daughter would be living with. He told me he was taking her and I went ballistic! I told him no she's not and that if he has a problem take me to court! He shut up right away and I guess he wasn't expecting that because I never got that crazy with him. I guess he thought I would do whatever he wanted but not anymore. After saying my peace on our marriage and finally letting him know how I feel and how he treats/treated me he started to cry and apologized many many times. He told me he doesn't want it to be over but he needs a break. I told him we can have one. But don't be upset if I move on without you. And once we go our separate ways I'm not looking back and neither should he. I still LOVE him deeply and as strong as I look on the outside every moment I'm alone I'm crying my heart out. But I know this is what needs to be done and honestly I feel proud of myself to finally stand up to him. I honestly hope this is what's best for us and this will help us in the future. And maybe one day once we've gotten ourselves together we can be a happy family and have a loving marriage. But I'll just let time do all the talking. Thanks for reading!