Have you experienced your boyfriend being aroused but too tired for sex?

I am only asking because I know he masturbates (which I am totally fine with) but I don't get it that I am around and we are just having a lazy day why he wouldn't act on it. I also realize that due to my past, I definitely have a hard tome separating sex from love. So I guess, I just want to know if this is normal and I should just get out of my head or if it something to worry about. It's also only been like 4 days since we had sex so I might just have a higher sex drive!!!