Boyfriend calls me names..?

I know this has nothing to do with babies & birth, but I am a mother of a 10 month old baby girl with my boyfriend. We have been doing really well, not fighting so much since the birth (but a lot during pregnancy), but today was really where I wanted to draw the line. But I need some advice. 
He started a new job today. He calls me on his way home from work-"do you know _____" I said yes, why? "Did y'all have sex" I replied, ".....4 years ago. Why?" He hung up. 
I wanted to be honest with him, and he knows my sexual history. It just didn't click with him that's who it was but when I asked him how he knew if he didn't tell him he said "I assumed, that's how sad it is. Everywhere I go I run into someone you've had sex with. You're a whore, always will be." He won't even let me talk to him & when I tried he threatened to punch me in the mouth. Of course I walked away & minded my own business!
An hour later he comes in the living room. "You're not going to cook anything? I worked all day don't you think I'm hungry? I guess that's just what hoes do." I was silent. "Still not gonna say anything or cook for me? Wow." I replied, "no! Why would I do that? Guess that's just the 'hoe' in me!"
I KNOW this treatment isn't acceptable, but I don't know how to handle it. I am not proud of my past. I feel bad for ruining his new job, he doesn't even want to work there anymore bc of this. I don't want to leave bc we have a great home & beautiful daughter together. Please give me your advice! Any would be greatly appreciated!