Canceling gender reveal

My husband and I were excited to plan a gender reveal for us and our close family.  Only thing is that my family has always welcomed him with open and loving arms, and his family has not done the same for me.  It's bad enough I tolerate his mother, but now his cousins and aunts and uncles have added to the drama.  Over the last year his uncles family has called me a liar, talked bad about me to his mom, and said we were jealous because they could get pregnant and we couldn't.  Needless to say, I don't have room for people like that in my life.  I have never done anything to them and have only supported my husband.  It's hard to deal with those kind of people because I'm not used to that.  My family is not mean spirited to others..
Here is the dilemma, I don't want to invite that uncle and his mean wife to the gender reveal, but I want all our other aunts and uncles there.  My husband got upset when I said this and it turned into a huge fight.  I think I just want to find out at the doctors and be done with it.  I don't want to fight and I don't want to have a gender reveal gathering anymore.  It's too exhausting.  His family already exhausts me with all the drama.  I don't want to add to it.