Baby daddy & my family... Help!

Erika • 💙5.12.15, expecting #2 💙 12.7.18
So I'm super close with my family. I also have a really good relationship with my fiancé. I have our ultrasound to find out the sex of the baby coming up in a few weeks, and id really like for my mom & sister to be there, but he is insistent that it be just the two of us. On one hand, I think it's sweet that he wants us to share that moment (he's a gruff type of guy, not the most sensitive usually, so this is sweet for him lol) but on the other hand, my mom & sister are a huge part of my support system & I know their feelings would be hurt to be excluded. 
Fiancé said he doesn't mind for them to be in the room for delivery. I think he's nervous about it just being him for that lol 
I just don't want to hurt anyone's feelings. I'm glad he's not fighting with me over them being in the room when baby gets here, but I don't want to hurt their feelings when I know they really want to be there to find out the baby's sex. Anyone else dealing with this, or dealt with it in the past? My tummy is in knots over it! :(