Husband's meltdown

Our baby is almost 4 weeks old and I think it just hit my husband today that life is never going to be the same again. I'm EBF so I'm up with the baby most of the time, including from 1am-6am (she's going thru growth spurt and colicky).
When my husband gets up he takes over while I sleep from 6am-9am. He's been really great with cooking and cleaning but when he's on baby duty, he gets really frustrated when the baby cries and has just been letting her cry it out because he's at his wits end. Today, I snapped because I was upset he was just letting her cry and he got angry saying that the baby only wants me, and how tired he was etc. 
I feel horrible because I have no sympathy for him! No shit, he's not the only one tired but this is what we signed up for. Please, does anyone have advice on how to deal with a frustrated husband? 
PS - we also made the decision that he's going to quit his job to be a SAHD. I go back to work Nov 1 so the baby will be 5 months by then - will she be better around him by that age? 

Glow Resources

Let’s Glow

Glow is here for you on your path to pregnancy

Glow helps you navigate your fertility journey with smart tools, personalized insights, and guidance from medical experts who understand what matters most.

25+ million

Users

4.8 stars

200k+ app ratings

20+

Medical advisors