Husband wants to try for another baby….I don’t want to though

We currently have an almost 3 year old boy, and I’m happy and content with how our lives are going right now. To be honest, I don’t want another child, I feel like everything is perfect with just one. I’ve been called selfish by some people for not wanting to give my son a sibling. Not only am I happy with just one child, but I dodged a huge bullet with my body and I’m very lucky to have “bounced back” looking better and thinner than my pre-pregnancy body. I’m scared having another child would definitely give me stretch marks and loose skin. So I don’t want to deal with that…but I also don’t want to make myself out as being selfish and preventing my husband from having another child (he really wants a little girl). Ugh