am I being selfish??

Me and my partner have a son together who is almost 5 months old, my partner would love another child at some point. I would love another too, However I had a horrific pregnancy, labour, delivery and postpartum that it's put me off ever getting pregnant again. My partner is really understanding and says that whatever I decide he will respect because it's my body but everyone keeps telling me I'm being ridiculous and that I'm selfish for making my son an only child. Obviously this is a premature question as he's only 20 Weeks old but everyone's comments and the guilt I now feel are getting to me. So my question is, was anyone else ever in this situation and overcome their anxiety towards having another? Thank you.