Emery?

What do you think of Emery for a boy? Both me and my husband really like the name but I'm afraid that people with think it's too girly. I've talked to my co workers about it and a few family members and they all really liked the name until we did it was for a boy and then they were just kind of like... oh.... and it just makes my heart sink. What do you think? Would you still name your kid something that you didn't think anyone liked?