Said no to me coming over, said needed “me” time

I’ve been dating this guy on and off for over a year now. We recently rekindled our relationship about a month ago when he reached out to me. We’ve been hanging out multiple times every week, texting every day, calling, and even planned a trip that’s coming up in a few weeks. I spent the night at his place last night and it was great. We went to bed at 3 am and got woken up at 7 am by a construction worker. He couldn’t go back to sleep but I did. He had work hours later. His cousin lives with him and I’m good friends with him and his gf. They wanted to hang out tonight at their house but of course I asked permission if I could first. The guy told me I can hang out with them but once he gets home from work he wants to go to bed and wake up early to go to the gym. I asked him if everything was okay and he said yes, he just wants some “me” time since he has a lot going on in his life rn. I expressed to him that I get nervous when he says stuff like that and he said there’s no need to and i should stop doing that, that he’s just tired. I get it bc he barley got sleep last night and had to work 12 hours after along with doing hw. But like I said, it makes me nervous. I decided not to go over to even just hang with my friends bc I don’t want to invade that space for when he does come home. He has a kid and when he’s not busy with her or the days he doesn’t have her, that’s usually when I do go over. I get that he needs his “me” time bc he’s always busy with this. He’s asked for this a handful of times before when nothing was wrong with us. I want to hang out tomorrow before I can’t see him for the rest of the week but I’m scared. What should I do?