Advice please..?

Ok so I'm just looking for some advice on my "crappy" relationship. We've been very on and off for 10 years, we have a young child together and were wanting to add to our family.  We have both had other sexual relations on our breaks witch has became a problem for us both since getting back together, I just feel so unhappy everyone told me not to take him back, we have a very bad history and now I feel ashamed and scared to leave him at the thought of all the unhappiness to come, but I so desperately want to break free and not waste any more time on this dead relationship .. My question is how can I get my head around walking away , since I've done it before I no the pain,heartache loneliness that's going to come before I can move on and it terrifies me.