Break up with someone you love?

For those of you who broke up with someone you loved, why did you break up with them?
I love my boyfriend, I love spending time with him, I love who he is and what he stands for, I miss him when he's gone and am happy when he's around.
But, aside from minor imperfections in our relationship, there is one hurt in our past that in our 3 years, it's never gotten easier to handle. It comes up rarely on my mind, but when it does, it's just as difficult to handle as before. I forgive him, I trust him, I don't judge him for the past. But it honestly still hurts to think about it, it creates an unwanted ache in my chest when I do.
I don't think I could ever leave him, but at this rate, I don't know when I'll ever be able to say "I do". 
Has anyone been in my position before? What did you do?