should I allow people in the room?

Sorry for the long post just looking for advice since I'm now 29w 4d. So when my boyfriend and I found out we were expecting we talked about letting people in the room. I was for having my mom, his mom, his grandmother, and aunt. I don't have any family where I live so my mom is flying out. Well he said he only wanted us and the doctors and nurses. So I said okay. Well a few weeks later I guess his family started (to me at least) guilt tripped him with the "well it's my first grand baby I should be there", so then he started to talk about letting them in. I then was like nope we're by going through this. I felt like they purposely did it when I wasn't there so they can be in. Well now I'm thinking this can be something to share with the since my mom will be here also. The main issue I have is that my mom and his mom annoy me. I mean drive me nuts! His mom had a fit when I said I wanted to do a 100% natural birth with no drugs at all. My mom just always try's to tell me what to do just like his. I feel like if I say just us I might regret it after and I won't be able to redo it. Anyone have any advice?