labor hijacking

So I've always said I just want it to be me and my bf during the delivery and recovery so we can bond and rest without visitors. He seems to be on board yet hasn't said anything to his family. I told my own family that we will let them know when we are ready for visitors and that a text saying we are going to the hospital is not an invitation. 
Well his sister made the comment the other day that she is going to be at a one day meeting out of town next month and said I hope it's not the day you go into labor. My thinking is why does that matter, you don't need to be around for the labor. My bf didn't say anything 😒. I just think his family has the wrong idea about what is going to happen when the baby comes. 
I feel like my bf is going to have a hard time saying anything to them because his sisters husband just went to rehab and his dad was just diagnosed with lung cancer with limited treatment options. 
What to do