He said his self esteem is low because of me

Me and my boyfriend have been dating for 4.5 years. I work full time, school full time, and have been grieving the death of my 16 year old cousin recently. My boyfriend works a regular 40hrs a week graveyard shift and that’s it. Im the one with the vehicle, I make sure we both have food every night, both get to work and both able to go places. Mind you I wake up at 7am for work and school and I still take him to work at 11pm. Most days I only have 30min-1hour to myself. My boyfriend is a sexual person but as of lately with school and grieving I haven’t been in a sexual mood. He will try to initiate ( 80% if the time it’s while I’m sleeping or trying to sleep) and I tell him no. What makes matters worse is, he will ask again, and again, and again after I said no. He will then roll over and stay silent making it weird and awkward. He said he continually asks like that so that he could at least get something out of me. I told him I am tired, I work, school and make sure everything else is done. I am also grieving a tragic loss of one of the closest people to me and I can’t always be horny and feel every emotion all the time. I’m emotionally and mentally exhausted. There’s days I don’t even feel like crying but I want to. I thought that my partner would be a little bit more understanding of my time of grief.

He even said that he’s been making sure he’s been a great boyfriend lately. Im not sure if he’s doing that with the expectation of sex in exchange? Wasn’t clarified.

We’ve been fighting about it a lot lately and he claims that I am breaking his self esteem. He feels like I don’t find him attractive. I’ve told him I do find him attractive and he should trust my word as his partner when I say that. I told him that I don’t find it fair that he is blaming me for his self esteem issues and basing his confidence on the fact that my actions are not meeting his expectations. All he said to that was “ interesting…”

I told him I’m sorry I’ve been making him feel that way but I don’t owe him sex. If his confidence is shot, I will help him regain it bc I love him and never want to see him like that.

Am I in the wrong? Am I supposed to be more understanding? Should he be blaming me for his self esteem issues?