Taking baby to see family before shots

So my bf was saying last night he's gonna take our son out whenever he wants to see his family (I'm 20 weeks pregnant) and honestly I don't feel comfortable with a bunch of people touching my baby before he's had his first set of shots. I have 2 other sons and was a single mother and I didn't let a bunch of family touch my boys til they had their first shots (only my mom and sister were aloud around the first few months and they knew no kissing or touching their faces) he has a large close knit family and I don't. I'm worried our baby will get sick because they're so vulnerable at those stages especially with RSV and other illnesses not to mention our son is due in the last week of October so it's going to start being cold. I want to set boundaries but I don't want him to think I'm being controlling I just don't want my baby to end up in the hospital.