Is it ok to tell someone they have too many kids?

Me and my husband have 4 kids together (blended family) and a baby due anyday now and people tell us we should stop. This is only our first child together and my 2 kids are the only ones that live with us. My step kids rarely come around because of their mother and the fact that they live a few hrs away from us as well. So?? Whats your opinion?

We honestly wouldn't mind another one after this one (a couple yrs from now of course).