3 month old

My baby is 3, almost 4 months old. He’s always been with me since day 1. Until he turned 3 months. He’s been visiting his granny & grandpa a lot more recently for a max of 4 days. There are 3 other people who stay at the house he goes to. MY granny just made me feel bad by saying he’s too small to “be passed around like that”. My sons granny works at a mine and is home for one week, and gone for a week so she asks to see him for a few days on her week off.

My concern is, IS he too small to be “passed around”? will he get confused on who his parents are? lol. Or will he get confused in general? He’s my first son, and his grannys first grandson.