Am I wrong???

I've told everyone I don't want any visitors while in the hospital...With my first I had sooo many people in the room with me I was young and my sons dad and I weren't together so i needed family and friends...16 years later I met the love of my life and we are expecting our baby girl any day now...i want it to be just him and I it's something I didn't get with my son...my 16yo Will come after she's born I do want him with us! Now everyone is mad and running their mouth to everyone in my family about how it's not right what I'm doing...its also flu season and that makes me really nervous...plus I'll be breast feeding and that is all new to me so I want privacy...am I being rude with my decision???