Boyfriend stays home sometimes?

I'm 18 and been with my boyfriend for 4 years. We now have a baby who's 8 weeks old. We don't live together yet but plan to by the end of the year. He's here most of the time, but there's days where he doesn't come over and stays at home (we live in different areas). Every week theres a day or two where he doesnt come. First he used to make up loads of excuses why he couldn't come, then when I said I was going to leave because I think he's cheating he told me it's because he likes a 'break'. Which really pissed me off because how can you want a break from your girlfriend and new baby? I don't get a break and he's the parent too. How can he want a break sometimes but he says he wants to hurry and get our own place. He says he's just at home by himself 'chilling', he says he's not cheating or hanging with friends. Last week i told him its over because i think theres more to it and he swore he would come every day from now on but today he stayed home again. He just cant see whats wrong with it. he says there's nothing wrong with staying home some times. I think there is something wrong with it seeing as we now have a baby and we are meant to be a family. I might sound stupid but i just don't see other families where the guy stays home or goes and stays out all day and night to have a 'break' from their girlfriends and children? It doesnt make sense to me. Am I wrong for expecting him to be here with us every single day? He says I'm overreacting for saying it's over because he's not here all of the time, is he right? I don't want to to be over so I need to know if I need to be a little bit easier on him and stop going so mad every time he stays home?