Can't get myself out of this relationship

So I met this guy almost 3 years ago. We hit it off and began talking and hanging out but eventually we realized we wanted different things. I wanted a relationship and he did not. I met someone else and dated him for a year and a half all while casually staying in contact with the other guy. We would say hey howre you every now and then and thats it. My bf and I broke up eventually and I started talking to the first guy again. We started hanging out and started dating after several months of hanging out and talking nonstop. It has been about a year since this started again and I want out. He is a wonderful person but he has issues. He gets jealous over dumb things and wants to hangout all of the time which I can't do because I have a kid and work full time. I see him when I can but when I can't he gets mad and starts arguments. Every time I try to break it off we just end up back together. He will say I destroyed his life, broke his heart, wants to die and I feel bad and start talking to him again then we meet up and end up having sex. This has happened like 4 times. There is a huge connection and attraction so I think that's why it's so hard. How do I break it off? He literally talks about suicide when I try to. I believe he would hurt himself too since he has tried killing himself several years ago. He is bipolar depressive so that's one of the issues he has. I just dont know how to get out without feeling like I will be responsible for something bad happening to him...