Can't orgasm while having sex with my boyfriend

This has been a huge problem for me since I've started having sex. I can't seem to orgasm with anyone except for my ex boyfriend and that was only one time out of three years!!! My boyfriend is very concerned and he feels it's him but it's me. Can you guys give me some advice if you have overcome this difficult situation. I can orgasm on my own with a vibrator or just me by myself. Please help!!!