Is “gringo” or “gringa” a derogatory term?

I work and live in a area which the majority is Hispanic my husband and I have been called these terms many times. Usually it’s when a customer wants to get my attention at work. I don’t react in any type of negative way, but I wonder if it’s a insult or just how a Hispanic person talks to a caucasian person. Is gringo or gringa a derogatory or negative term to use?