When do you know #5?

When did you guys it was time for #5? We have 4 and our lives so SOOOOOOO full. But, I just don’t feel done. My last pregnancy was rough on me. How did you guys decide?