My husband grabbed my arm in an “argument “

We are in the middle of moving house so tension is high. My husband finished work and came downstairs telling me a list of things we and particularly I need to get done. I told him to stop panicking and went upstairs to start boxing things up as he asked. He followed me up and angrily told me to stop telling him not to panic and how he has taken time off work because there is so much to do and frankly I haven’t progressed the work enough (we had 3 kids downstairs including a newborn). I really wasn’t up for his shouting at me and started to walk away, he angrily grabbed my arm - the stair gate was between us so I pulled away and he couldn’t follow. I said to him that if he physically hurts me again, I’ll call the police. Then walked away. Now, I’m nursing the baby, the big ones are in the bath. He comes into the room and starts shouting how I made him feel with that comment and my mom who was downstairs might have heard it. He said I have no idea of what physically hurting someone means and if I ever accuse him of this again he’ll file for divorce. I’m stunned. I couldn’t even react as he just kept repeating the same thing, shouting for about 5 minutes until he just left. I asked if I can say something but he was gone and I was still nursing the baby. What is happening? He never apologised for hurting me. I probably overreacted with the “police” comment but grabbing arms in an argument is not acceptable for me. But, is it really enough reason for threatening me with divorce? I don’t even know how to talk to him about anything after this. Please share any advice. I really don’t want a divorce and I can’t even see how any of what happened would justify it.