Newborn and grandparents/ husband

Our baby is about 2 weeks old and sleeps all day during this phase obviously. My husband’s parents live on the first floor of the house and he gets upset the baby is downstairs playing with his grandparents. It’s irritating one his mother calls to manipulate him about the baby not being in her presence.. not sure how much playing you’re going to get out of him while he’s sleeping. He goes down when he’s awake and the grandmother helps when she can so it’s not like she’s isolated from her grandchild. It is very annoying tho she calls to tell my husband the baby isn’t with her then he calls me to yell at me like who the hell am I not the child’s mother? Their are grandkids out there that don’t have the luxury of seeing their grandparents on an hourly basics like my parents live in another state. Should he not be more understanding with me versus his mom?