Family or friends mean to baby

What if you're in the situation where a family member or friend is mean to your baby (teasing them until they cry and won't leave them alone) and nobody else bats an eye at it and you as the mom gets angry removes the baby from the situation and doesn't allow it to continue and doesn't allow the baby to be around the person that's mean to them. And then family and friends call you being over protective or oversensitive.