Birth Plan Drama

So, I am in need of the experts here in the August babies group 😊
My husband and I made our birth plan back when I was only a few months along and it has stuck. We are both very firm in what we want. We have decided that we do NOT want to inform anyone that we are in labor with the exception of my mother. I am sensitive (almost overly so) and we both worry that having a bunch of people in the waiting room will stall labor because I will likely grow anxious about everyone think I am taking too long or getting bored. 
In order to combat that, we decided we would have our baby totally alone and then make phone calls when we were ready to receive visitors. We mentioned this at a family (his side) dinner last night and we were actually told "No." They basically said that it was an event for everyone and if we didn't call we would be taking away their joy at seeing the first grandchild. We brushed off the comments and changed the conversation but their message sill stood. They expect us to call so they can sit in the waiting room while I labor. 
My question is this, would it be wrong to simply ignore them and do what we planned and take the fall out? Or should we try to convince them of our plan? The bottom line is, I am not calling until after I am holding my son. I just want to avoid as much drama as possible. What do you guys think?