DV marriage

So I just recently got divorced. I have a daughter who is 4 and a son who is 18 months. My husband put a gun to his head when I said I wanted to leave, in front of both my children. His family blamed me for ‘pushing him to that point’, even when it originally happened and I called the police. I ended up leaving and didn’t have anywhere to take my children so I left them with him. I carry a lot of guilt but I knew I wasn’t safe and he hadn’t physically hurt our kids. I went over to his house every day to be with my kids while he was at work but it began to progress into full blown panic attacks just knowing I had to go back there to see my kids because I was so frightened, so I found a place to live and got a restraining order and custody. I was a stay at home mom, no license, no GED or high school diploma, no car, nothing. I was a teen mom and my husband worked. So I have had to start my life over from scratch. So long story short I lost my place to live and had to give my kids back to their father and drop the restraining order because I didn’t want them to be homeless. Now I have a new boyfriend who is helping me put my life together, I feel extremely guilty leaving my kids with their father though. In the divorce I couldn’t fight for custody because I don’t have a place to live or an income so we ended up agreeing to joint custody but they live with him and I get every other weekend. He is constantly putting me down and telling me I’m not trying to be their mother because I don’t have a job yet and because the every other weekend thing hasn’t been working out well. On top of everything, this is in the last two months they have been with their dad, i just found out I have Mono which makes me sleep all the time and I’ve been getting secondary illnesses like strep and tonsillitis multiple times. So he has been verbally abusive continuously throughout everything and won’t stop. I’m about to the point of giving up because I’m at a loss of what to do.