Relationship problems

So me and my husband have been married for a year and we have a three month old son. I'm a SAHM and my husband works full time. I do my best to take care of our son and I even take care of him during the nights and don't wake my husband (even on the weekends). For about two months now my husband seems to hate being home on the weekends. He tells me our son drives him crazy because he cries. I try to explain to him that he's just a young baby and to give him time to get older and he'll enjoy him more. He seems miserable with his life 😞 it seems like he would rather be at work than with his family.  I try to get him to talk about it but he says he likes being home but I know he doesn't. I just wish he would enjoy our son. I do everything I can to make having a baby easier on my husband but it doesn't seem to be working.  I guess I don't really have an exact question...more of a rant but I just don't know what to do? I feel like eventually he'll just get tired of us and leave. Anyone else have a similar problem? I feel like it's just me 😓