I don't know how to feel over this racial issue. Thoughts?

I am half black and half white. All throughout my life I have heard people talk bad about black women. This has come from white people and even black men. Whenever I tell the person I don't appreciate them talking about half of my race like that, they say you don't count because you are light skinned!? It feels like they are saying I should not feel offended by them talking bad about half of my race because I am not fully black.