Alcohol & Sperm

My husband and I have been TTC 4 years. His semen analysis is good except the morphology is borderline. 3 years ago morphology was 0% and now it's 3%. Lately he has been drinking alcohol more than I would prefer. Typically he'll have a drink once or twice a week, no big deal! But lately it's been everyday. Sometimes 1 drink, sometimes a couple. He's not getting drunk or anything. So my question is, should I be concerned as far as his sperm and TTC? Or should I just let this go. He gets mad when I say something about it being almost everyday lately. Thoughts?