Advice: Leaving a relationship

Hi! I’m 22 years old and I recently left a relationship with a 26 year old man. I have known his family for around 10 years and we started dating a year and a half ago. We lived together for majority of that. He had gotten out of a extremely toxic 3 year relationship (with a child) prior to our relationship. I also was in one for 8 months, but I had no children. Throughout our relationship we’ve had a lot of positives and negatives. On the positive side, we went to 7 different states together throughout our relationship, he owns his own home, we have a dog together, I grew a relationship with his sweet daughter, we tried new things together, etc. But, on the negative side, he had a lot of anger/temperament issues, called me out my name on many occasions (a bitch, dumbass, stupid, etc), cornered me (never hit me though), yelled at me in front of my friends & his family, talked shit about me to his family/friends, kicked me out, threatened me, threatens to break up w me etc. A few weeks ago, he ended the relationship, and for the first time I got up, packed my things, and left. He proceeds to tell me that He didn’t actually want me to leave and that it was stupid that I left, etc. Mind you, we never told eachother we loved each other. He told me he couldn’t love me and that we had to work on our relationship before he could love me. But, after I actually left, he proceeds to tell me he loves me & wants me back. I suffer from anxiety & depression & i have really lost myself within this relationship. Yes we did so many cool things, but at the end of the day, my mental health is suffering. Do you guys think it’s wrong that I left this relationship? He also is getting custody of his daughter & lost both of his parents within the last two years. It really makes me feel like shit but I haven’t been able to be there for myself at the end of the day. I just need some sort of advice.