My 4 month old doesn’t want anything to do with me lol

I spend everyday with my 4 month old son.

Now it almost seems as if he wants nothing to do with me. He smiles at me once and then gets mad or cries, as soon as his dad comes around it’s instant smiling and laughing, he’s fine being up with him, etc but then if I do it, I just get starred at or he cries sometimes when he’s up with me.

He does this with his grandpas, he will smile at them and be fine with them but then the grandmas do it and nothing, one smile and then he gets upset.

I feel so sad about it because he’s my first baby, my little boy and I feel like he doesn’t want to be near me at all. Will this change, is he going through a phase? I’m confused.