Husband

I want some help to understand if I am just crazy or is he being selfish

So to begin we haven’t been on the best place for a while. He’s been drinking heavily at first then we had our first born. He still been drinking heavily for her first year. After that he decided to quit his job and turn his life around. I was happy he’s finally changing his ways. We got to know I am pregnant with a second child. To be completely honest I was not overly happy about it. Anyways so fast forward few months we started to really fall apart. I tried talking to him and all I got was he’s not happy because we don’t have sex anymore as often. Or as good as we use to. Well I mean I was pregnant wasn’t drinking and he was still drinking more then he should. So we kept falling apart and now we have a cone baby he went on the fishing trip with his friends on our 5year anniversary also our second baby is only 3 weeks old. After birth I cannot really control my emotions or felling I haven’t really been myself. So anyways I talked to my mom and his mom and decided it will be good for me to go out and see him and air my head a little. It’s a 3 hour drive. So I come here he told me he will be back from the lake at 9 I get here he txt me and says he will leave the lake in an hour or so. Which will be 10 by the time he gets back will be 12ish. I am crazy to get upset and wanting to go back?! I left my 2 year old and my 3 weeks old at their grandmas which I know loves them unconditionally but I am their mom they need me and he literally doesn’t give a shit about it. Please advice