I HATE his family

Clo
My boyfriend and I have been dating for nearly 3 years. We both will live apart with our families. We both grew up and still live with different lifestyles. I guess you could say I come from a "middle class" family and he comes from a "lower" class. My family owns a house, each of us has our own car, and I have been very fortunate to never worry about basic necessities. My boyfriend on the other hand, has had a pretty rough life. His mom and her boyfriend are alcoholics, everyone in his household are heavy smokers (including him), and they struggle to pay bills every month. 
My boyfriend is "disabled", so he gets a social security check each month. I feel like his family makes him pay a ridiculous amount, he pays $550 of the $750. There are four other people living with him that work full-time, and there are 9 people total living with him. 3 of the 4 are always buying alcohol and weed. The 4th one doesn't really have her priorities straight either. Instead of buying diapers and formula for her kids, she bought a sex toy and a new phone. My boyfriend isn't great with money either, and spends his extra money on video games instead of practical stuff like new boxers or socks. So I end up buying him stuff all the time. I always have to pay for dates, and he's never gotten me a birthday gift or anything. I understand he doesn't have a lot of money, I just wish he would pay for dates sometimes. 
Another thing that irks me is how the family treats the children. His sister has 2 kids. The oldest is almost 4 and can't speak full sentences. They never read to the children. They rarely let them play outside. Don't get me started on their diet. The kids are always sick. I've never seen them healthy. Plus they live in filthy living conditions. 
I feel like this problem comes back to his family. I think they take advantage of him. And He doesn't do anything about it. He refuses to move out because he doesn't want to "abandon" his family. I really don't know how to deal with this because his family keeps wanting more and more from him, and me. They've asked to borrow money for cigarettes, food, baby formula, and diapers. And I have a hard time saying no, so I agree and sometimes never get paid back. My boyfriend and I have argued on several occasions about his family. I'm at a lost on what to do. In a perfect world, he would move out. That's all I want. Because I can't stand his family. 
UPDATE 1: I say "disabled" in quotations because I'm not sure of the correct terminology. 
As to why I'm with my boyfriend, it's because I love him dearly. I met him when I was a senior in high school and he lived right down the street from me.  I had recently broke up with an abusive boyfriend, then was diagnosed with endometriosis, went through a terrible depression, and he really helped me through it. It's the little things he does for me, like bringing me a snack when I'm hungry or giving me a massage after a long day at work. He tells me I'm beautiful with or without makeup, in sweatpants or a miniskirt. He loves me a billion out of ten. And he would do anything to make me happy. I never had a boyfriend that treated me so well. That's why it's so hard for me to decide to stay or go.