Husband wants alone time

My husband recently ask for 2 days off from work to have a 4 day weekend, we dont get to spend a lot of time together so I did the same. Well it's a day before our 4 days weekend and he tell me "I didnt want u to take these days off with me, I really want to be alone."

Some background on us: My husband is a wonderful man. He's funny, smart, handsome, and a very hard worker. This isn't a post about cheating, I trust him completely, and know he would never do that to me. We've been married for 6 years no kids yet (ttc). About 3 years ago he got moved to night shift. I work days which means we dont see each other much. We both have Sunday off which is nice, but other than the 1 day during the week that he's off and I rush home to see him before I have to go to sleep, that's pretty much the only day we spend together (a few hours of it mostly)

The problem: He basically told me he didnt want me to get upset but that he didnt want to spend time with me, he wants to play Xbox since he doesn't get to play much. But how can I not be upset about this? I get home to an empty house 4 days a week and go to sleep by myself pretty much every night of the week. I'm alone all the time. I really just want to spend time with him.

Am I wrong for this? I told him I didnt want him to feel forced to spend time with me which is true, but I also said that he can't expect me not be a little upset about it. It really hurt my feelings.

If you read all this thanks for sticking with me, I just needed to vent.