who's has success with low sperm and conceived naturally?

Sarah
DH and I are currently trying on and off for 2 years now we finally have help from the fertility specialist everything was completely fine on my end but DH is undergoing treatment such as clomid to help his sperm count he was a body builder so he did have substance abuse .. He has been off it for 1 year and 8 months the endocarnologist said that he will put DH on this medication as it works for many of his previous body builders...his blood levels are finally looking sharp his motility and morphology jumped from 2 % to 32and 1% to 36% his sperm went from under 1 million to 3 million in 3 weeks and 6million in another 2 and a half weeks .. The specialist said that we should still keep trying but the stress of trying with low sperm .. It's just such a let down  and I tend to put to much pressure just my personality hence why I'm asking do you think it's better to just wait till his levels are up or to just keep on trying ?