3 year old is starting to notice different skin tones... Advice please.

Julia
The other day my 3 year old son was playing with our upstairs neighbours' daughter in the courtyard. (We live in an apartment complex.) And after he came back inside he was telling "Oh, I see the brown girl! I want to play with her again!"
And I was shocked. I told him, "It's not nice to call her brown."
And he asked why, of course. Because that's what 3 year olds do. 
Well, I couldn't really give him an answer. Because I didn't know how to explain it. I just told him it wasn't polite and to please not call people brown.
He didn't understand. He said: "But she IS brown."
Which was the truth, but I don't feel like that's right to say.
I really need some advice. I'm so afraid he's going to say the wrong thing to the wrong person. I understand he's young and doesn't mean it that way. Kids are just blunt and not always polite. But I'm really worried he's going to seriously offend someone. I don't know how to handle it. I'm half filipino so I can tan well and my son is a quarter Filipino but he has such a fair complexion, you wouldn't be able to tell. He hasn't noticed a difference between myself and him. I'm scared that someone will think I'm either teaching him about this or... I don't know, really. How can I teach him about the looks of other people in a positive way that a 3 year old will understand? Without being racist.