Am I overreacting?

Trying to keep this short: we are TTC and have a sex schedule. Two nights ago I shower and shave and get all ready for it. Get in bed in my nightie and hubby is watching tv/on the phone, no initiation whatsoever, so I fall asleep. Wake up an hourish later to him initiating and I'm conflicted about whether I want to. But it feels good so I go with it and it ends really good. Flash forward to tonight (2 nights later and time for sex again): hubby gets home late from work and I'm already ready for bed. I spend some time with him first but he's on his phone/watching tv so I eventually go to bed without him. I'm super tired after working two long days this week. Is it wrong of me to be upset that he woke me up an hour and a half later when he had an opportunity before I fell asleep? He claims he didn't want to come to bed as soon as he got home but since when does sex mean you have to go to bed? Just want to know if I'm crazy for being mad since I did enjoy two nights ago.

Update: He knew it was a sex night, he knew I was tired, and I spent almost an hour talking and cuddling with him in his recliner (but only halfway having his attention) instead of going straight to bed. We had been sending dirty texts earlier in the day so there should have been no doubt as to what I wanted. I'm more frustrated that it's turning into an every night thing almost like he doesn't want to put in effort while I'm awake. I also explained that I was tired and he made it out like I shouldn't be upset because he didn't want to as soon as he got home. Maybe that explains a little better.