Touching yourself during sex?

last night my boyfriend and I were having sex and I decided to touch myself, which took a lot of courage for me by the way because I have low self esteem, because I'm sure most of you know it's hard to have an orgasm just by vaginal sex. I've never had an orgasm because I don't really masturbate, and again, low self esteem. But my boyfriend didn't like that I did that because HE wanted to make me feel good. I don't really know how to feel about this because I was kind of hurt because he knows I'm insecure sexually and I know a lot of guys get turned on by that. Opinions/advice? Am I wrong? How should I approach this?