Still didn’t work!

My husband and I have been trying for over a year. He has a low sperm count. 5mill. Then they suggested clomid. He took it for three months straight and re tested. We went today and it’s down to 1mill. I don’t understand why this is happening to us. They say 85% percent of times it helps increase male sperm. Unfortunately we’re in that 15% that doesn’t. The doctor is saying IVF might be the best route, but shiiiiit!! 10-15k is no joke. At this point I don’t know if giving up is easier.