he hates me because I'm white

My boyfriend hates my family because they're white. He's always making comments about white people and cracking jokes at my expense. I was molested as a child by my brother and he's always saying "that makes sense, you white people are all incesty" and today he called me a cracker, this is nothing new. But for once I stood up for myself and he just hung up the phone because "he didn't want to fight again" and he's always saying I give him headaches. When we're good we're great but today just sucks. And he's always saying he can't be racist because he's Hispanic but oh shit this hurts.