Advice Needed....

So I'm a 1st time mom with a 6month old.Me and dad have been together for 7 years.Going to try and make this straight to the point as possible.....Before baby came we had moved into his family members home(we are tenants) which turned out to be rat infested. I told him that I was NOT staying there with baby. I will be staying in an empty apt. my parents own. Most would say why wouldn't he leave and move out. .. the in laws also live with him and are on fixed income cannot afford "the real world rent".(on list for elderly housing) They also don't want to end things on bad terms with family etc. Now he won't stay with me because he's finishing parole and now is when they try to get you "out of line" to take you back.He had told me before baby was born the relationship would not work out if we are in 2 different homes/cities.he never understood I did this for the safety of my child because it was a rat infested house. In my heart/mind it absolutely could this was a temp. move until we 1-could get our own place or 2 finish parole come live with me.Well here we are 6 months later and my feelings for him are indeed not the same.He has given me hell about picking him up n having days with him/over nights since baby was 3 weeks old.Apart from this on "his days" everyone I go to pick him up out of work he's never with him. baby is with his mom or older sister (14 his daughter ) or not home/outside smoking weed (which I have nothing against but feel like your kid comes first). I feel like I get no help from him. Never could I just get up n go do what I need to because I have my son, mean while he has that"freedom" n time to hang out.Financially does nothing n says he had a lot of things to pay can't even go 1/2 on daycare. So I certainly have not been feeling the same towards him. I feel like a single mom,doing everything alone. I am torn because I thought baby would bring us closer because we both wanted it. n I don't regret my son he is my miracle child (I'm a cancer survivor / lost his twin) but I never would have thought that my 7yr relationship would turn into this from 1 day to the next. I love him n would like to be with him but I just don't see it being fixed because he sees nothing wrong. What do you think of this? I feel stuck. Do I let it go or hold on n try to make it work?