please, I need advice!

Please someone tell me if im in the wrong, my fiancé and I had our own apartment, we saved up money went and got a car, but eventually couldn't afford the payments, so we are moving back in with his mom. We got into it pretty bad today because he wanted me to stay at his moms and do laundry while he went to work. But I didn't want to, I have REALLY BAD anxiety, and his mom does not talk to me, I don't know why, we've been together for 4 years and she still will not talk to me. We don't have our room together yet, so I don't really have anywhere to hangout, except for down stairs with her and I'd just be really uncomfortable for hours until he got home, so I asked if I could go to my friends house until he got off work. So I don't have to be alone with her and eventually end up having a panic attack because of my nerves, I just can't handle being in uncomfortable situations. But he got mad at me for asking, saying that I needed to get the laundry done, but yesterday I cleaned his moms whole house with no help, (it's a huge house) and started on the laundry. Last night, instead of him helping me finish the laundry, he came home from work long enough to tell me he was going to hang out with friends. He sleeps all day, doesn't wake up to spend time with his daughter, then goes to work from 2-12 (midnight) so she's already in bed by the time he gets home. I keep her all day everyday, without a break and if I go out and hang out with friends, she always comes with me. But it's rare that I even hang out with anyone because I only have one friend. I ended up blowing up on him because he gets to go to his friends whenever he wants, but the one time I ask, I have to stay home and get stuff done even though, he does NOTHING. Besides sleeps, goes to work and hangs out with his friends. Am I in the wrong for blowing up on him? I mean, I know everyone needs breaks and needs time with friends, but why do I never get a break? I'm going insane. I get a babysitter once every 4 months or so, it's very rare. And I'm constantly at home 24/7 what's it gonna hurt if I hang out with a friend for one day? Until I can be more comfortable with staying with his moms. Please someone who understands where I'm coming from with the anxiety give me advice, I don't know what to do.