Relationship isn't the same

Me and my SO have had 2 miscarriages so when we found out my little peanut was healthy we were beyond happy. My son is everything to me and I wouldn't trade him for the world. But, ever since I've had him (6 weeks) my relationship is NOT the same. I'm so frustrated with everything he does. He works 7am-7pm and his only off day is Sunday. I'm so grateful he works so hard for us. But, since Sunday is our only family day I always plan a fun day with baby.

This Sunday he wakes up and says he's going to his moms house to help her work around the house (she's older) and that he wants me to bring the baby by later. But WOULDNT give me a time 🙄 So I sat at home doing NOTHING because I was waiting for him to tell me the come. Finally at 8pm I call him and see WTF he's been doing. He says he's been "hanging out and drinking" with his mom and best friend.

You would think he would want to be with his son on his ONLY off day, right? This is only one particular thing that got under my skin. I could keep going with all the crap he's been doing but I'll stop here. I just feel so alone and feel like he's out having a good time while me and his son are missing him. I'm seriously debating whether or not I want to stay with him 😢 we've been together for 5 years and he's NEVER EVER been this way. We used to never argue and be so in love. I thought our son would make our relationship strong. But it's done the opposite. What should I do? Is anyone else having relationship problems after baby? Or am I the only one?