Am I in the wrong for this?

So I’m a stay at home mom, I can’t drive due to Epilepsy, I pretty much stay in the house all week with my son. I don’t know anyone around where we live because we just moved here. I literally feel like I’m going crazy because I never leave.. my fiancé thinks I’m in the wrong for wanting him to come home straight after work and hangout with us on the weekends because I don’t think it’s fair that I never get to do anything and All I do all day is sit around and wait for him to get home. I would love to get a job, but we’re honestly saving more money not paying for a babysitter plus I wouldn’t have a ride. We don’t even know anyone around here who could babysit for us. He thinks im naggy for wanting him to stay in with me but I honestly think his job is a luxury because as much as I love my son, I would kill to go back to work. It drives me crazy being home all day, and not having anyone to talk to. My son is only 1 and can’t talk yet. We live in the country, the closest thing to us is a dollar General store 3 miles away. It’s honestly getting so old that he can’t see it from my point of view. Its becoming an everyday type of fight now. He’s been having work friends over on weekends since “I won’t let him do anything outside the house” and they just eat all our food, get drunk, and work on cars well I take care of our son and do the same shit I’ve been doing. I feel like he thinks just because he makes the money, and can just be in control of everything. I’m at my breaking point and am about to say fuck it and leave and he’s not understanding at all where I’m coming from.