Is my baby mad at me?

So. I work overnights 9-9. I drop him off at my parents at 7 and put him to bed there. When I pick him up in the morning he refuses to look at me for the first 20 minutes. He will look at everyone else but me. I will even pick him up and get close to his face. He finally will break and smile but it's so weird. Even when my husband is home when I go to work. When I get home he does the same thing. He's nothing but smiles when my husband gets home from work.