I fucked up

Me and my SO have been having a hard time in our relationship. I have a hard time trusting him because of him cheating in the past and on top of that, I'm 18 weeks pregnant... once a night every weekend he goes out to drink with his friends until 4:30-5AM and I feel like we have so many issues in our relationship, that he could take the time to have nights to work on us, but he doesn't... he chooses to go drink. Well, today I was irritated because last night I stayed home with his daughter while he went out for "a" beer with coworkers... I wanted to have a family bonding night or something. We never get to.. 
Fast forward to later In the day, he's pissed because I'm pissed... we were in the car to go to his grandmas house and he keeps talking shit to me about how I'm ungrateful and when I say something he makes that stupid annoying baby voice to mock me and that pissed me off so bad. Before he started talking shit, I was quiet and just ready to get to where we were going, enjoy it and leave! But he continued to talk shit to me and do the baby voice thing, taunting me, and I lost my shit and hit him/scratched his face...
Now I feel terrible for putting my hands on him. I also feel like I could not take the taunting and have so much anger built up inside of me right now, I just didn't know how else to let it out. 
He took me home and now he's pissed at me. I don't know what to do or say, but he says he's done with me. I would rather work things out but at this point, I will not try and beg for him to stay in this relationship. I just feel awful for touching him that way... 
UPDATE:
I know I was wrong. I'm not saying I wasn't wrong and yes it would be wrong if he hit me. I don't need you bitches to keep rubbing in that putting my hands on him was out of line because clearly, I understand that.