Should I stay?

Sort of a long story, but I'll try to be to the point about it.
I'm friends with an ex of mine, he suffers from depression and is always stressed and tired because of his job and other things, which tends to interfere with our friendship. We're also long distance, which doesn't help. 
We got into an argument because he always thinks badly about himself, and apparently doesn't like it when I say good things about him since he doesn't believe it, he thinks I hate all these things about him when I don't. He isn't great at communicating and just gave up after a while when I said I didn't understand some of what he was saying. 
The day after I asked him if he actually wanted me in his life, and if he was just trying to make me leave him alone, since my friend said that's what it sounds like. He wouldn't give me an answer and just said that he didn't know what to say since he was just tired and busy and depressed. He said it was up to me and that I could stay if I want or leave if I wished. 
I love and care about him immensely and don't want to leave, but after everything my friend said about it I don't know what I should do. She thinks if he "doesn't care" if I leave then I should. He's miserable and pushes everyone away unless they keep making the effort to stay like I have, and as far as I know, he doesn't really have any other friends. He does make me happy, but when things like this happen I don't even know how to feel. 
Should I keep trying or leave him be? 😞