DON’T PICK HIM UP WHEN HE’S CALM

So my family comes over a lot and I have a 4 month old. He rarely likes to be put down he’s getting better at entertaining himself and playing with toys. But I swear he’ll be in his bouncer or his pack and play babbling and playing to himself and every single time someone will pick him up. Even if I say “don’t pick him up if he’s calm, or if he’s not crying.” They still do it and when he starts to cry from them annoying him they hand him to me when I’m busy. 🙄 I’m a young first time mom and they all act like I don’t know what I’m talking about. It’s so annoying!!