Our girl name

Tiffany
So my hubbys middle name is Ray, as well as his younger brother that got killed in a car wreck several years ago.. So the middle name is Rae :) we were going to go with Layla Rae but ended up not liking it. So my hubby mentioned naming her after his brother :) which im totally ok with. I love my hubby, and his little brother was very important. Our 6yr old is named after 2 grandpas. So both of them have a special name. Her 1st name will be Hunter(which is unisex i googled it lol) and middle is Rae(for daddy). I think its incredibly adorable but when i tell people they get all nasty and pop off thats a boys name, poor baby girl :-/ like really.. 
Is it that bad?