Selfishness???

So, my dh and I decided to try for another baby. This will be our 4th, and  I've already heard from the few people we have told, that we are being selfish and are taking away from the women who don't even have one. My question is how is this even possible?? How does my fertility affect another woman's?? I can understand the jealousy she might feel, given how I have 3, and she might not have 1 yet, but how am I being selfish?? How am I taking away from her??