Husband doesn't seem to be happy

My husband has two weeks for paternity leave and he's been with me for 10 days now. He's been a great help physically (doing groceries, laundry, dishes, taking our 5 year old son out) while I focus on our newborn, breastfeeding, cooking and cleaning. It's been super difficult, for both me and him. We both planned this baby and knew our lives would be more difficult since there would be more responsibilities. The only difference is that (I feel) I am embracing the difficulties with a positive attitude. My husband on the other hand always has an angry and upset demeanor. As soon as things go uphill for him he literally gets angry and won't talk to me and won't look at me. His tone towards me is cold while when his mom comes his tone changes to super sweet....He admits he loses his patience with everything and especially our 5 year old.. he's constantly wanting to go out to blow some steam off and it's worrying me that he's happier away from the family. I've tried to talk to him multiple times about this and each time he gets defensive and says " you always have to make this about yourself". And I'm thinking, what? I'm telling you how I feel and that means I only think about myself? I asked him to hold off his outings till baby is at least one month hold so I can get a feel of her routine (after all, isn't paternity leave a time for family to be together and help each other out vs taking this time off to go out and play volleyball twice a week for hours on end and attend friends bday parties?) idk what to do anymore but I'm tired of crying and being upset every night. I just feel he's making this a toxic marriage. Positive advice is appreciated.