Scented body wash/detergent?

My LO's father is military and just got back. We have a five-week old baby. He uses body wash and detergent with fragrance in it and that worries me a bit with him holding the baby a lot. Am I being overprotective? I don't want to bother him to change everything unless it's a legitimate concern.