Gingers

I have naturally red hair. In the winter it is VERY red, in the summer it's usually considered "strawberry blonde". Regardless of the time of year, I always fall under the same category: Ginger.
For many years now, people have called me a "ginger". I'm going to be honest, I find this word offensive. Usually the word is not used as a kind, light-hearted nickname, it's more often a name used in jokes that are often hurtful.
My question to you non-red-haired-folk is, have you ever called your red headed friend a ginger? Have you ever stopped to wonder if the name is offensive to them?
To the fellow red heads, do you find this term offensive or hurtful?
I'm ready for everyone to be done labeling people based off of their physical characteristics. Let's stop handing out names that pertain to hair color, skin color, gender or ethnicity.