how to handle people inviting themselves to the hospital

Ah
a bunch of extended family and old friends keep saying they'll be at the hospital when baby is born but i never invited them. Mind you, these are people that I don't speak to often. I know myself and I'm way too high stress for a crowd of people to be there right after i push a baby out. I wasn't even planning on announcing his birth til we were home and somewhat adjusted. 
The plan has always been for my mom and SO to be in the room when he's born. my brothers and MIL will come later. but as for everyone else, idk why they think they can just show up. Should i tell them that I don't want visitors or just let them find out the hard way after we're home with baby? is anyone else having this issue with people? I personally think its rude to invite yourself to the hospital after a baby is born. it's not really a public event.