Afraid of my boyfriend

My boyfriend has anger issues and they have gotten pretty bad. His mom told me he’s always been like this since a child and that he has always exhibited bad behavior. He often gets upset over little things like when I forget to do laundry or when I don’t cook because i’m sleepy. He does these things himself too but not as often as i do. Yesterday we got into an argument because he was mad at me because i went to my mothers house and didn’t come back at the time he wanted me to. I came home maybe an hour later then what he wanted. I didn’t do it on purpose I was having a conversation with my mom and we were having fun together. He was so mad at me for this he even called me on my way home and yelled at me. When i got in the house we started to argue and he had punched a hole in the wall. I wasn’t too phased because he’s done this before but after that he started to yell at me some more and then he smacked me in my face. I was pretty caught off guard by it. He’s never hit me before. He also said to me that if i listened to him, he wouldn’t have gotten mad and he would’ve have hit me. I ended up just going to bed because i was pretty hurt by this. In the morning he apologized and got me some gifts and swore he’d never do it again. I forgave him but I still feel a little bad that i made him so upset.