Leave Him?

I have been with this guy for two months. He is really nice and sweet but... he says things when he's "jokeing" that hurt me, like "I just fuck girls when you're busy" or "you taste shitty that's why I never go down on you" they hurt my feelings and he just goes "you know I'm joking babe" I find it harder and harder to stay, but he was my first and I love him, but he doesn't ever take me out or even want me over to his place. Not to mention he knows I was sexually assaulted, yet he talks about forcing me to have sex and letting his roommates do things to me without asking me... I know I should leave him but... I can't.