Bed with SO

So my wonderful boyfriend who I love dearly always complains about me not coming to bed with him exactly when he wants to. I just want a few minutes by myself to cool down and get little stuff done. He doesn't understand.