3 month old hates being held by anyone but his dad and me

Recently my 3 month old son does not want to be held by anyone else but his dad and me. He literally screams as soon as they pick him up. And he doesn't like to be put down most of the day either.  While it's flattering that he only wants us to hold him, I  have to go back to work in a few weeks and I'm worried his sitter won't want to watch him if he cries all day. Anyone else going through this or have any advice?