The pros and con of my relationship

Pros: kind words, in close proximity, in love with me,  makes me laugh, whenever I'm scared he calms me down. Whenever I'm sad or stressed/need advice he helps. Supportive of my decisions. 
Cons: doesn't like to go out (what we argue about most), selfish, when we argue he talks down(says you're stupid etc) he's depressed. Doesn't know what he wants to do in life. Thinks he's not good enough for me. Hardly texts or calls when we are apart. (I initiate it all). 
I feel like there's more to add but I just don't know. Yet these are the pros and cons of my relationship. So many people tell me to break up with my BF because I don't deserve the issues that u have with him yet no matter how many people tell me to leave him it's so difficult to give up on us. He can break my heart 70 times and stay with him yet I hurt his feelings once and he's ready to leave....why is it like that? 
Is it really this hard to let go? 
Should I honestly go? 
I need advice ladies! 
Ps, I'm in college (idk if this helps)