How soon is too soon

My husband and I suffer from male factor infertility. We’ve been trying for 3 years, have done 3 rounds of <a href="https://glowing.com/glow-fertility-program">IUI</a>. We went to a reproductive endocrinologist and she review his SA and said his count was way too low, and <a href="https://glowing.com/glow-fertility-program">IVF</a> is our only option. My question is how soon is too soon to start buying baby stuff? We are planning our retrieval for May/June.