Unknown Cause of Sexual Fears

I am a sexually active woman, 20 years old. I have had sex for around a year now with a man I trust deeply and have no issues with when it comes to intimacy. However, I have a strong issue with him eating me out or touching my vaginal area with his hands. He does not get upset with me whatsoever, he is gentle and caring and never scares me or picks at me for it. I don't know why I am so terrified of it. I can touch myself no problem and feel fine but when someone else does it I flinch hard and pull my knees tight closed. This has been an issue for awhile now. When I have vaginal exams they have to hold my knees open to even get near me. All I can remember or think of is when I was little, a man touching me between my legs and feeling uncomfortable. I think I was at a doctors office on an exam table but I'm not sure. I keep having that image in my head. How can I relax enough to allow someone to touch me?