I can't be the only one....

I'm not sure how to even word this, and I'm a little embarrassed by it, but here goes! So I've never orgasmed from sex. Masturbation yes, but not sex. Any time my s/o even gently touches my clit my legs start shaking and I get panicky and push him off. I can't help it! I know I can't be the only one that happens to, and I wish I knew how to get comfortable with it! Any advice?