Tonight is the last night my daughter and I will live with her dad. Am I making the right decision?

So tonight my daughter's dad and I got in a big fight because he had been being a jerk all day. I dont want to get into details because I'm honestly so drained, but as I'm feeding our daughter- he called me stupid and a whore. I lost my shit. I said do not EVER talk to me like that in front of my child again and walked away. I cant even look at him. This is about the 4th time he has done this. He wont even apologize and now is guilt tripping me for saying I think we need to be done.

Is it bad of me if I just write him a note and leave with the baby tomorrow? I have tried to leave so many times and he talks me out of it, or it escalates. I just cant handle it anymore. He's so terrible to me, but makes it seem like it's all my fault. I'm not a perfect person or spouse, but i dont think I deserve that...