He wants a baby shower, I don't

So I am 21 weeks now and my fiancé wants us to have a baby shower, his co-workers have already said they want to throw us one, as well as so many of his family members asking when we are having one. The problem is I have severe anxiety and some other issues and I get very uncomfortable around large groups of people, especially people I don't know well. The thought of being any kind of center of attention is terrifying for me. I have explained to him why I don't want one but he's still pushing. This is my 4th child, his 2nd, our 1st together. I have no family here at all, they're literally on the other side of the country so that makes it more difficult for me too. I'm just looking for advice. I don't want to be selfish because I know he is excited and would like a shower, but it's truly scary for me. Any thoughts?