Husband had the flu, should we abstain?

My husband had the flu with a high fever a week ago. I am concerned that his sperm quality could have been affected, resulting in an unhealthy baby if we concieve with "fever sperm." My fertile window is next week. Wondering if we should abstain next week if a high fever could affect sperm quality. If we are going to conceive I want to make sure we have a healthy baby. Anyone know if we should abstain? Or just go for it?