Hit me

My boyfriend got mad that I called him stupid in front of his brother. After he left he swung and hit me in my arm. Then I got mad and hit him back and he hit me harder. This is the 3rd time he hits me in the arm during this week for the first time ever. Is this a sign he is abusive? Is this my cue to go? He promises he isn't abusive and would never hit me seriously but I know it eventually will get to a point where it will be serious. I told him he has no right to put a hand on me. He said well I shouldn't have name called him especially infront of his family.