Confused on this one, guys

Let's say the average ejaculate contains 1 million sperm, for ease of discussion. And a man's body needs time to replenish. Let's assume 1 day for a full tank. If you do it three times in one day, wouldn't it just put fewer in you that time, say .3 million or so , but more overall if you were doing it more frequently? Folks talk about low sperm count if you do it too much, but isn't it just for that specific ejaculate? More sperm in your body is more sperm, right? The quality of a specific ejaculate would go down but not the specific sperm , right?

What element of this am I missing? I know this is a funny post but I'm trying to learn more about male issues the more time goes by...