Not the answer I wanted

My husband and I had our first appointment with a fertility specialist today. We have been trying to conceive for over a year. Turns out my husband has a very low sperm count. It’s a little over one million. The doctor told us that most likely our only option will be IVF. We talked about that and broke down costs and everything and I’m just so overwhelmed. Financing interest rates are ridiculous. How did everyone do it? Did you all finance? All of our savings are supposed to be going towards building a house so now I feel like everything we have planned for is just no more. I just don’t know what to do. Any advice would be appreciated. I’m just feeling really down right now.