Deciding on number of children

S💍
This might sound crazy but how do you decide how many children you're going to have, once the process has started and infertility is not an issue. Before we started I wanted 4 and my husband was thinking 1. We're at 2 handsome little boys. 
Life is manageable, fun and all around great. I'm at the point where my body is yearning to be pregnant again. This all got me thinking, when/why do ppl decide enough children is enough?? I know we have extra time, energy, and money but should we have another child and stretch those resources further when right now there's more than enough?