Slim Chance To None

So even before we start trying my DH has it in his mind that we won't get pregnant because he was told when he was 16 (now 33) that he had a low sperm count and that it would be hard and almost impossible TTC. My question is has anyone else had to deal with this and any word of wisdom to go along with it?