Don’t want kids around my son

I have a few friends who have vocalized that their kids want to meet my son who is 4 weeks old. I honestly do not feel comfortable having my son around children yet because I feel like my friend’s kids are sick all the time and I don’t want my son to get sick. I also do not feel comfortable with the idea of children handling my son.

Is it rude/unrealistic to want to wait for him to meet any other children until he is 2-3 months old? How can I tell my friends I do not want my son to be around other children without coming across as insensitive?