hitting?

my boyfriend is the sweetest most caring guy ever i truly mean that when i say it too, he loves me so much and would do anything for me. i know he has a side of anger issues, but i don’t think he’d ever do anything to hurt me even if he was angry and he’s assured me he would never hurt me too. he’s very sensitive and emotional too, anytime i am hurt or he does hurt me by accident he gets so worked up about it and feels awful. we usually get pretty into it during sex and get aggressive and rough, yesterday he slapped me in the face twice during it just part of the act, but i cried and stormed out afraid of him. he followed me and cried and felt so awful that he hurt me. he apologized repeatedly to me. my question is should i be worried or should this just be a misunderstanding?