Newborn and grandparents

I’m due in a few weeks and really considering not having visitors right away or letting family hold him for the first few weeks. What are your thoughts?