Taking newborn (4 wks old) in public

What’s everyone’s thoughts on this?

My grandpas 85th birthday celebration at a restaurant and my son will be 4 weeks old by then. A lot of people say not to take him in public that soon, and to wait til he’s a little older/built an immune system up.

I really want to go but don’t know if it’s a bad idea.

My plan would be to take him but just not let anyone hold him. Would that be okay?