Husband taking wife's last name

I am wondering if anyone can give me a reason (other than sexist ones) that would back up why a man cannot take his wife's last name. I have talked to men and women about this, and women tend to see no problem with it, while men say it is wrong because it is not "traditional" or "manly."
The reason I ask is because my husband and I are considering doing this very thing. I have not changed my name yet, because I was a single mom before I married him, and my son has my last (maiden) name. 
I do not want to change my name, and leave my son as the odd one out, especially since we are having another child. We just want our family to all have the same last name. 
Adoption is the preferred option, but we worry that this may take a long time, if it ever happens at all (do not want to post the personal details).