describing color of skin using flavor

🍁🌿🌲🌻
My SO thinks it's racist to describe color of skin using flavors, where I disagree. Would you find it offensive/rude? Please no arguing. This post is to discover other's point of view. 
Ex:
- Pale skin: Vanilla
- Tan skin: Caramel 
- Dark skin: Chocolate