Don’t know how to feel

So my baby’s father and I have not been seeing eye to eye since the moment I became pregnant. He moved in with me in the middle of January and I became pregnant right after that I found out February 11th. He had moved 2 hours from his home town and he still hadn’t had a job which was understandable because he just moved in with March came around he still didn’t have a job around the 2nd week in March he got a job for $10 an hour working barely 15 hours a week. I was upset he chose such an awful job but he assured me it was temporary and only to make money for the time being while he looked for something else...well April came and he was still at this crappy job, I had just got let go from mine and he actually found a job that paid well and would give him full time hours. He got really upset one night at my friends boyfriend and went and punched his car repeatedly. He ended up having a hairline fracture in his pinky finger and the doctor gave him a whole cast. With me just losing my job and his job saying he couldn’t start there now I was livid at how childish he was to have punched his car out of anger. I left to visit my grandma 10 hours away because I had just felt so depressed and upset at the situation I was in. My stay with my grandma ended up being almost a month but I was still making money doing so cleaning for her that nobody else would do. While he still wasn’t working because of his pinky. We got into numerous fights while I was gone because I’ve paid for everything in the relationship since he moved in January every date every bill it was me. And he said I shouldn’t be upset because money shouldn’t matter in relationships. But I’m thinking about my baby and how we are gonna provide for him. I have since found a job and I’m still paying for everything myself, but he moved out while I was gone and said it was best so he could work on himself. I have no friends or family around me and I’m literally so depressed I’m alone all the time and I just hate this feeling. He says he wants to be together but he’s not ready to move back in because he needs to “find himself” since all I did was nag at him about money. At this point should I even let him come back because he said he wants to come back eventually and be a “family”. Or should I just end up dealing with legal stuff because he’s a lost cause?