Baby cries when anyone else holds/talks to him

R
Got a question about my 5 month old. Ever since he was a few months old, whenever anyone has tried to hold him or gets in his face to talk to him, he breaks down crying and doesn't stop till he's back in my or my husbands arms. Doesn't matter if I'm in the same room or not. My husbands family thinks that holding him till he stops crying will get him used to other people but he literally cried his eyes out till he was so exausted and just whimpered till he was given back to me eyes soaked and all. My question is...is this the right way to socialize my baby?? I didn't like how he was just taken from me, brought upstairs out of my sight for over half an hour being forced to be held. I don't feel this is the proper way to get my baby used to other people. I want him to when he's ready. He's a sensitive baby and gets scared easily. Strangers, loud noises, unfamiliar voices. I didn't feel respected as his Mother either. People were just trying to take him from me and pass him around which was really stressing him out. I understand these are his relatives and they just want to hold him and I would love for them to be able to w/out my son crying but not asking me to hold him and keeping him in a room upstairs away from me is not ok with me and I'm not letting that happen again. When I visited my friend, it took my son 4 days to get used to her and then he let her hold him when HE was ready. That made more sense to me. Once she could hold him,he was looking up at her smiling. He wasn't forced into it. Any advice on what else I can do to help my LO w/this or am I doing it right to let him be ready on his own  time?