Why is my boyfriends brother always talking badly about me?

I’m in an interracial relationship with my boyfriend he’s 26 and I’m 27. His brother is 30 years old and is constantly talking about me behind my back. In person he pretends to be nice, but my boyfriend tells me once I leave he makes racist comments. He recently talked about how he thinks my hair is fake, which I think is weird, I’ve never met a man who obsesses so much over a woman’s hair, that’s very feminine. My boyfriend told me he said it last week and today during his birthday party. My boyfriend and his brother live in the same house so avoiding him is difficult. Their mother owns the they house live in. She moved out after their father passed away so they pay her rent each month, while she lives in a separate apartment across town. My boyfriend said that he and his brother got into an argument during his birthday party when his brother started talking and he stood up for me. The problem is that he still expects me to come over to his house and hangout with him like nothing happened knowing I’m going to run into his brother every time I visit. I told him I didn’t feel comfortable visiting tomorrow after hearing what his brother said and he made it seem like I was punishing him for what his brother said but I think he needs to be more sympathetic towards my feelings. He says that by not visiting I’m letting his brother win and he feels like I’m breaking up with him. How should I handle this I love my boyfriend but his brother is becoming a nuisance.