26 weeks and my significant other wants me out

We’ve been together for over 2 years and I’m due in September with our second son. My bf suffers from depression/bipolar. He’s difficult to live with it’s at some points but his temper once he gets mad he kicks me and our son out. I finally told him I’m not moving out again. He only sees things his way and doesn’t look at the big picture. Do I just let go and move on with my 2 boys. He also is a fireman and just got sent home for a back injury. He can’t help with our 1 year old? Am I the one that should leave?