My Bf Grabbed Me While We Were Fighting

We were fighting because he had barely talked to me in two days for longer than five minutes and we live together, and I have bad anxiety so I thought it was my fault and we ended up arguing instead of confronting it. But he kept turning it around to how stressed he is and how hard things have been for him and that that's why he hasn't talked to me, and the way he said it was like I was stressing him. I decided I didn't want to talk about it, and when I walked off he grabbed me and pulled me back and I kept telling him not to do that and he said it was because I wasn't listening to him. He was raised in an abusive household and said the same angry temper his dad had was passed to him, and I've seen him fight himself not to be like his dad. But last night he was just like that.

I gave him an ultimatum, that if he ever grabbed me like that again we were done. He broke down after that and then came and found me and gave me a note saying he doesn't know how to apologize enough, that he hasn't been himself in months and that he'll never forgive himself. He's been off for months, I'll admit, after we almost drowned together in a river and never got to deal with the trauma of that. But he said it was like he woke up again when I told him not to grab me like that, because that's what his mom used to say to his dad.

My heart is telling me to Just love him and give it time, because he's never been this way, not in 18 months together. I just never thought I'd have this problem with him. I've always had the instant reaction that if a man puts his hands on a woman in a way that hurts her, and she tells him not to, she should leave without question. And his intention wasn't to hurt me, but he did. There's no marks or anything, but there doesn't have to be. The thing is, I really want to wait and see if he'll actually change. If he doesn't return to the man I fell in love with, then I'll go, but it's conflicting because on principle I should've started packing as soon as he did that. But now we want to work through it, and I'm scared he'll just go back to how he was before he "woke up". What should I do?