Fantasised of another guy during sex?

I've had this problem with my boyfriend for a couple of months now where he can't make me orgasm. 
He made me orgasm once and that was when I touched myself during it so I guess that could count as my orgasm & not his. 
However, tonight I was riding my boyfriend and I began to think of my new colleague that I work with and how attractive he is. I began fantasising while I was riding him and anything about him that would turn me on, I climaxed! 
I feel GUILTY because my boyfriend thinks it's from him, when I'm thinking of another guy. Well it was my boyfriends dick but my thoughts were on my work mate. 
I feel extremely bad, because it was somewhat my first orgasm through sex, and my first ever orgasm with him. Should I feel bad? Why did this happen? I never could orgasm on my own and I did thinking of another guy :( 
I'm so in love with my boyfriend, I wouldn't replace him for the world.