Latex Allergy?

My boyfriend and I always use latex condoms when we have sex, but this past time that we had sex with a latex condom the left inner side to the entrance of my vagina has become slightly irritated and I'm not quite sure why. I figured it may be a latex allergy so I tried doing some research and there really wasn't anything specific and I've never had this problem previously. Anybody have a similar experience or insight? I've taken Benedryl and even iced to help the irritation.