Work/personal relationships are complicated

My boyfriend can be a bully sometimes. He likes to rag on me in front of my coworkers because I didn't go to a private school like all the others did. I went to a public college and he didn't go at all. Several of my coworkers never went to college. Counting me, only 5 of us have. Secondly, he is always ragging on my accent and where I come from but today he crossed the line. Sure it's fine to rag me a bit cause I do get him back, however; where we work, we all eat lunch together, family style around a table. I was engaged In a conversation with another coworker and so was he. I heard my name and looked up and said Huh? I figured that was perfectly acceptable. He died laughing and said the the only other female coworker we have, see she is special needs. I was mortified. It was in front of all my coworkers. I stood up, threw a lunch bag at him and proceeded down stairs and avoided him all afternoon. 

Later on, one of our coworkers grabbed me and said, he really does love you ta know. I don't know how to feel. My feelings are hurt but he did apologize. He can be a real sweetheart and a very good man when he isn't being an ass. I'm just wondering if I haven't bitten off more than I can chew. The reason this is his from our coworkers is because I currently hold an apprentice position and he is afraid that the owner will fire me if he finds out, because the owner had the hots for me too.  I really don't know what to do. Any thoughts ladies?