Boyfriend wants a 'break'..Am I right for being mad or am i overreacting?

I have a boyfriend and a 6 week old baby. We don't live together yet, we plan to in the next few weeks when we find somewhere. However he's here all the time with us, well most of the time. There will be 1 or 2 days out of the week when he doesn't come to see us at all. His excuse is he wants a 'break' and a decent nights sleep. Before he admitted this to me, he always made up excuses which didn't make sense why he can't come over. I was accusing him of cheating and was going to leave, that's when he said it's just because he likes a break. He says he wants to hurry up and get our own place, but as I've said to him, how's that going to work when he wants breaks from me and the baby every week?. I've told him that it's his baby as much as mine, he doesn't just get to have a break and leave me to look after him by myself. He says he's tired, I told him don't you think I'm tired? Dont u think id like a break? I do the night feeds every night I have done since he's been born, even when his dad is here he just wakes me up to feed him, he just makes the bottle and then goes back to sleep. To me its strange that theres always a few days out of every week. I don't know if it's because he's with friends, or another girl, or it really is because he just doesn't want to be with me every single day because he does still text me non stop when hes not here and he only has WiFi at home, but it's still strange why he wouldnt wanna be with me and his baby instead of being at home by himself. Another thing is he smokes weed and because i don't allow him to smoke anywhere in my house, he likes to go home to 'chill in bed and smoke'. i asked him is it because you just want time with your friends, he said no he doesn't see his friends anymore. Ive asked him if hes seeing someone else loads of times, of course he said no. To me those are the only reasons he could be doing it, but no he says he just likes a break. I said so if it's none of those, its basically it's just because you don't wanna be with me every single day which i just dont understand if he wants to live with me. Yesterday we made plans to go out, i had a baby sitter. Instead he goes home without me, and when i ask him if i can go over and he said no, he just wants to go to bed. To me there's just no excuse for that. There's no excuse not to be here with us. I get that being together 24/7 isn't healthy but we have plans to get our own place, we have a baby now, if he wants to be a family in our own home and settle down like he says he does then aren't we going to be together every day anyway? Am I overreacting for going on at him every time he doesn't come over? Or should he be allowed a few days 'break'? Ive said im leaving him, he tells me im just being stupid, its only one day and theres nothing wrong with that. He said it might look like hes doing something wronf but hes not, and that i need to stop thinking things and just trust him. Would I be completely overreacting to leave him over this?