No holding baby please

How do you politely tell family members no when they want to hold your baby? It’s impossible to be out with my in laws because they just want to be holding my baby the entire time. We both have separation anxiety so when they do he doesn’t cry, he just looks at me scared and everyone just says “Aw, he’s so content with grandpa or grandma” and really he’s acting how he acts when he’s nervous and afraid. He doesn’t always act this ways with others. He laughs and and giggles and talks to my mom. He loves her a ton. But she also knows when to hand him back to me. So next time they take him how do I say no politely? It’s also not as much of a question as a warning. Like “Come to grandpa” or “Can I hold the baby?” But without even getting an answer coming to take him.