Need some advice

yesterday my boyfriend said he didn't want to know anything about me for a while and that he needed his space, so he didn't speak or contact me for around 5hrs. We did end up talking about this thing I was mad about later that night, so i guess everything's good. The thing is, the only times I ever get mad if he tries to bail on me is when we've already made plans to spend said day together. for example, if for some reason we can't do what we had planned on doing that day, he'd be like "well since we won't be doing this, I think I'll just go do this instead." so, instead of "we can do this instead" or "we can just stay home doing whatever" he chooses to do something that doesn't necessarily include me. Other than that most of the time he visits on days we hadn't really planned anything at all, so he could be using these other days to do whatever he prefers. I'm not saying that I don't like it that he comes often or anything, the problem is that when he goes on about needing his time, he makes it sound as if I was the one that made him come over all the time on days he could've done whatever else he does on his free time. Am I wrong for getting mad/hurt?