baby looking to the side

Chrysta
My bubs is 3 weeks old and just kinda looks to the side and not at me. I thought they start looking at u by this point. He looks to the side at absolutely nothing. Maybe white walls lol. Should I be worried?