How to handle visitors after baby is born?

For all the moms to be or moms who have babies already, how are you planning to or have handled visitors once baby is born? We are thinking of restricting visitors at the hospital to just our family like parents and siblings. Then having friends visit us once we are back home. I'm a little bit of an anxious person and it really scares me that too many people around could make baby sick or it might just be overwhelming for my husband and I. We have friends that have casually mentioned that they will be in the waiting room in the hospital when I'm in labor. I'm already worried enough about the dynamics of our families being there since I have divorced parents who generally don't get along. My SIL has two young boys who seem to always have a cold, flu or stomach bug and I know they won't think it's a big deal to have the boys around as soon as baby is born. I am just hoping neither will be sick when baby arrives so it's Ambon-issue, but if they are i want to tell her that I would prefer the boys wait to meet baby until they are not sick. I guess I'm just wondering if you all are planning on having friends and family at the hospital or if you are going to have a period of time after baby arrives where you have only family visit.