Should I give up?

Hubby told me the other day that he was going to stop taking a tablet to help increase sperm chances. In his 20s he was told the likelihood of him successfully having a child were very slim cos of naughty body building habits (that he stopped in late 20s) being in his 40s now and we're trying cos he (and i) want a bub. Side effects of tablets are nasty pimples on his arms mostly, hence why he wants to stop. Being 4mths now of us TTC... I've tried to encourage to get his count tested just to see if there's any increase chance... if not then there's no point in trying if I can't conceive. Thoughts??