Racism...

I've been with my SO for 5 years. Hes mixed race asian and white. His white side of his family is very conservative and very obvious they are racist towards Hispanics(me) and black people. They have never said anything terrible to me in my face but they will tell my Bf things when I'm not around. For example they will tell him "I hope you're using protection, Mexicans dont make good babies" "are her parents illegal? How did they get here" "a lot of Mexicans during the holidays go get women pregnant" tbh I cant remember some, but it hurts my feelings that they think that way towards me and people like me. I've asked my boyfriend what his response is to those comments and he said he doesn't say anything. He told me he doesn't say much because he doesn't want to argue with people bc they were raised in the south and that's how they've always been. His whole life his white side told him that mexicans and black people are terrible, all of his friends are Mexican and black! But honestly I think it's wrong that he doesn't defend me or my people. What do you guys think? I want to bring this up but I want to make sure I'm not being sensitive.