Help .. am I wrong long post

Ok so my bd and I live in separate houses and we have a baby together he’s 8 months . he lives with his mom I go to his house on the weekend so he can see our son . Sometimes I go Thursday-Sunday or Friday -Sunday depends. Anyways he has free time every day I feel get to go wherever whenever etc whenever he’s not working and he only works Sunday-Thursday 11pm-8am. & I work 6-2 6 days a week on those weekends I feel like it’s my time to relax or can a little bit like he does during the week or help me more and not go with his friends or go to the store or have a lil smoke break he will leave me in the house with our son and just go do something like every weekend we come stay it’s a reason he’s leaving the house. Last night he said “ ik your gone prb say no but my uncle ask me to help them move their cousin furniture into her apartment” it was like 11 at night I’m like wtf then I say no you can go with them tomorrow cus wtf but I just say ok so I could avoid the many reason on why he should go..but am I doing too much by feeling some type of way about him not being in the house with us at least one whole full day or for wanting at least a hour or 30 minutes of doing nun but myself when I say I don’t want to watch him rn at the moment bc sometimes I don’t I’m tired I watch my son 25/8 not tryna sound like complaining but damn sometimes I need a small break and I don’t feel like I’m asking too much especially from you when you’re his father I would expect you to be like oh okay it’s my weekend anyway its my time to you know take some of the mom duties for a little bit and don’t get me wrong I don’t mind him going places bc I understand it’s the weekend but every weekend is a no for me .. I need opinions fast ..