should I be mad?

My hubby came home drunk when it's supposed to be my fertile week. He says he really wants to have a baby and we have fertility tests in a couple of weeks. He says it's my fault as I never tell him when I want a "fun weekend". 16 months ttc. Surely he should want to keep his fertility as good as poss by not getting drunk? Am I being too hard on him? Just had a hard month as I worked 8-midnight most nights and also taking my dad to hospital as he has cancer so it was a ttc write off but we had a week away where we got things back on track. He was upset that we haven't bd'd much in that month but I had explained why at the time. Now he says he's the only one trying but that's just not true. So done!