Sex with my boyfriend.

I feel like I always initiate it. And we both have full time jobs so we're never really around each other much, but every night for the past 4 nights I've said hey babe, I'd really like to have sex tonight and he's all yeaaaah woohoo lol. But when we get home from both of our jobs that night, he decides to go into his brothers room and play video games until 3 or 4 in the morning. I have a sleeping condition to where I have to have a certain amount of sleep to function correctly. So last night when he finally came to bed, around 4, he tried to have sex with me. And I just wasn't having it. I'm tired of asking for sex, and I'm tired of waiting and waiting and waiting. Am I being crazy?