Boyfriend's Self Esteem…

Okay… so basically my boyfriend blames me for him not feeling confident. I try my hardest to be the perfect girlfriend, but it seems as though it's never enough. I know he has self esteem issues, I compliment him every chance I get, only for it to spiral into an argument.
He also believes I'm not truly, physically, interested in him. Since we're long distance now, sometimes our schedules clash and we don't get to be as sexually intimate. Sometimes, I'm just not feeling it when he is…
I don't know if it's because I'm not being as attentive, the distance… 
It hurts that he feels this way and I honestly feel like a horrible person.
How do I show him that I actually do appreciate his body?