Why so CRANKY?!?????

So I need yalls opinion on something... FTM and ever since me and my husband have been home with our newborn we only hold him when he needs to eat... and than change him and than put him down in his swing or rock n play. He falls asleep immediately with no fuss. He stays asleep until he gets hungry again. He's such a good baby and a good sleeper. Vincent is 2 weeks old and I've started taking him to grandparents. He saw them on Tuesday and today (Thursday). Every time he comes back he is super cranky. I tell my parents this and than I find out that my mom most of the time that he is there holds him for most of the time he's there. He is there for about 5.5 hours. And I found out my dad picked him up while he was sleeping to place him next to him. And now every time that he comes back from my parents he wants to be held otherwise he is cranky and won't sleep well like he normally does. Do you think I should say something to my parents? Or is this normal for him to want to be held this much? He was 5 weeks early and was born at 35 weeks. I'm afraid my parents are starting to get him into a habit that may not benefit him. Like why pick him up when he is completely comfortable and asleep? Just let him sleep.