Am i wrong here

Every time we take our one year old to my in laws and he starts crying when they want to hold him they say he’s crying because “we don’t bring him enough”. This really gets on my nerves. We live down the street from each other and try to take both our kids to see them once a week and that’s all we hear. Is it bad that it bothers me?