To tired to have sex

My partner always says how tired he is and that he can't have sex (mind you we are ttc)

The other day I was absolutely tired, I woke up to go toilet and then went back to bed. I was almost asleep and my partner came over and starting playing with my boobs and started kissing me. I said I was tired and going back to sleep and he just kept going.

I gave in and we ended up having sex.

I can't help but feel pissed off and annoyed about it.

He is always too tired and when I still try he pushes me away.

I ask if we can have sex more and he says his to tired etc.