Have I gone too far?

Hi all. This is a difficult one to post because I've had a hard time accepting what I done/am doing. Here goes.. I was with my ex for about 7 years from a young age. It was a terrible relationship in that he was abusive, he cheated ALOT, we were on and off more times than I can count but I was young and naive and everytime he said sorry I believed him and took him back. I tried to leave several times but he wouldn't let me go and wouldn't let me leave the house or car until I agreed to stay in the relationship. I had no self esteem or confidence and was convinced that was my life forever. Then one night after him openly telling me he was going to look for someone else as he thought the grass was greener(!!!) I plucked up the courage and walked away, without looking back. HOWEVER, here's my problem. During the last 2 years of the relationship (and trying to leave him several times) I met someone else...and I fell in love with him. He treated me well, was so so nice to me, made me feel good about myself, and loved me back - more than anything. He knew my situation. And we started seeing each other. I'm not proud of it, but I finally felt wanted, and loved. Anyway, fast forward 5 years and we're now together with a baby and still very much in love. But I'm having some past issues come up again. I feel insecure, I'm having trust issues and I'm anxious about him finding someone else because I don't feel like I'm enough to keep him interested - I wasn't enough for my ex so what if it's the same now? I've accused him of seeing other people (even though the only place he goes is to work) and texting other women (he barely touches his phone other than to play games!) And I've really hurt him to the point where he says he doesn't know what else to do and can't spend his life like this. Am I just damaged goods now?! How can I stop this madness?!! PLEASE help me!