Husband never home. makes me feel like a nag

My husband and I have two babies (Almost 2 &6 months)  He got back from a week long guy's vacation two days ago. Last night was his first night home and he went to bed at 6pm because apparently he was exhausted. He's at work today and just texted telling me he won't be home until late because he's going over to his friends house to hang out. When I asked him not to since I want to spend time with him because he just got back he said I was being a nag. Am I really being that ridiculous? I mean I think it's a little bit excessive to want to go out with his friends again after being on vacation with them all week. Would you mad? I almost positive he's not cheating because he's not really that kind of guy. I've talked to him about always being out with his friends and he tells me that I just want to keep him confined to the house. Our 2 year old is always asking for him but he's never here because he always has somewhere else to be. He got home from work at 530pm yesterday complained about how tired he was for 30 mins then went to bed leaving me to do bedtime alone. I'm so drained. Emotionally and physically. He makes me feel crazy or controlling everytime I ask him to stay home and I just don't know what to do anymore. He doesn't even run his plans by me he just tells me what he's going to be doing and I'm sick of it. I never get to go anywhere because all my friends and family are out of state.