Help!!

My husband and I have been trying to get pregnant for 2 years, I have 2 children from a previous relationship and 6 months ago we decided to get tested, I came back all clear but he has a seriously low sperm count, at first I was sympathetic and told him not to blame himself but now I'm getting fed up, the doctor gave us loads of advice to help and the biggest one was for him to stop smoking, he hasn't done anything the doctor recommended. I'm trying my best to be supportive but I'm starting to resent him. Does this make me an awful person?