Why can't I orgasm?

I have only had an orgasm once before in my entire life, when I was being fingered once (years ago with a now ex-boyfriend)

This is incredibly frustrating because sometimes I get really close sometimes while having sex with my current boyfriend and he tries so hard, and it just doesn't happen or it goes away. I have tried masturbating by myself, him going down on me as well as him fingering me. Am I doing something wrong?