Who should go to the 20w ultrasound?

Tomorrow is our 20 week ultrasound and of course my husband is planning on going with me. However, I also invited my mother and my mother in law (so she doesn't feel left out). I feel like anymore people would be too much to fit in the room and part of it is done vaginally. Because of this I did not invite my dad or my husbands dad. Now my husbands dad is throwing a fit about not going and I feel terrible, but I just don't feel comfortable with him in the room. Has  anyone run into this problem? I just don't know why he would even expect to go?