Trying for a third...soonish

So my April baby is our second. We’ve been on and off about trying for a third, and we just don’t feel “done” with our two. We have a 5.5 year old son and 9 month old daughter. We’re thinking of trying after our daughter turns one as I will be 36, and it would be nice to have two similar in age since there’s a gap between my son and daughter. Trouble is, EVERYONE we come across automatically assumes we’re “done”

This is because I had terrible HG and preeclampsia in my last pregnancy. Is it foolish to be afraid to try again because of “what people say?” I know we should do what we want to do to complete our family, but people can be so stressful. My OB supports another pregnancy with proper care.