Racist for wearing black? 🙄

An African American guy said I was racist for wearing black all the time. What the fuck? How is that even remotely racist?