Using A Unisex Name

We are naming our daughter Charlie, we absolutely love the name and don't plan on changing it no matter what. But the negative feedback we are getting just drives me nuts. Wasn't sure if anyone else is dealing with this or not..