Ftm-visitors after baby..

My husband and I are having a hard time agreeing on how to do visitors after our son is born. He'll be taking two weeks vacation so we have time together with him. I know family and friends will want to visit but we don't want him being passed around and held by everyone! What's a nice way of going about this? We really can't say no to everyone to visit. His grandparents want to see him and close friends. If I say no to one person I feel it's only right to say no to everyone.. Help please!! This is stressing us😣 I don't want to seem like a bitch but want it to be clear that he's not going to be held by everyone.