Clomid has reverse effect on husband

My DH has been on it for 3 months. His fsh had some budging room, and his count was low. He's been taking 25mg of clomid, daily for three months. We went yesterday to get he results.
Effff clomid. It has been horrible for him, his count is the lowest it's ever been and his hormones are off the charts high. 
So frustrated! A waste of three month and now even longer to get things normal for him again.
We planned on ivf anyway, but it's been explained to us now, that ivf is the only way we will ever conceive. We kinda knew that, but hard to hear, for sure.
Anyone else has clomid for his for DH?