Opinion on this?

Five years ago I was engaged to a guy. About 9 months before we were supposed to be married, I developed a slight crush on someone else and I ended my engagement. The "crush" and I were just friends, and I never cheated (we hooked up a few times much later, after I broke it off with the fiancé) but I felt as if getting married was wrong, simply because if it were the right choice, I wouldn't have had those feelings about someone else. So here I am, 5 years later, happily married to someone completely different and I bumped into my ex. It was pretty clear that he's still harboring a grudge for me having left him "for some other dick", as he said. At the time, I didn't feel like it was a trashy or shady thing to do, I just felt like I was ending a relationship that wasn't right to begin with; he, however, seems to think that what I did was just as bad as if I had cheated, and now I feel like a shit person. What do you all think? (And yes, I know none of this matters and it won't change the past, but I'm having some self-doubt about the kind of person I've been, and I just want someone else's take on the situation.)