updated 3/19*- hospital visitation- am I being too harsh?

Lavinia • Proud momma to 4 babies. Expecting baby #5! Also Proud momma to 3 angels above us
I'm due in May with our rainbow baby and our first to make it through the pregnancy. My husband and I both decided we want it to be just us during labor and just us for the first few hours after having our babygirl to have that special bonding time. My parents understand and live just down the street- said to just call when they can come down, she'll be their 7th grandbaby. His side of the family is a different story. She'll be his dad's first grand baby and his moms third, but first granddaughter. His dad&girlfriend, and his mom and 2 sisters want to be there when she comes out so they can all hold her. Which I understand from their standpoint but at the same time that takes away from our time as husband/wife and mommy/daddy to bond with our daughter. I'm now being called controlling and harsh for not wanting anyone there until after she comes and we've had time to bond with her as a family before she gets handed around to everyone else. Am I being too harsh? My husband wants us to stand our ground and says we're the parents and making the decision we feel is right for us. 
Update- went went to my mother in laws yesterday and I'll be damned, she said exactly what we don't want to happen. She said as soon as our baby is born she gets to hold her first even if the placenta isn't out yet. His sisters then chorused in that they were next. My husband just gave me this look. We were both happy we decided in advance not to tell anyone until after she comes. I'll be damned if my daughter gets handed around to everyone in the world before her own mother and father get to hold her and bond with her 
Update 2- my husband felt he would be lying to his mom if we didn't tell them before hand our plans so he texted her saying we'll tell them when I go into labor but we want no visitors for the first hour. She texted back "wtf" "what do you mean?" So he responded with we just want the first hour to ourselves to bond with maddie then we'll call you to let you know you guys can come. After that she now will no longer text him or answer any of his calls.