Irritation

For some reason, after my SO uses a condom with me, my skin gets really irritated. I don't think I'm allergic to latex because I eat bananas, avacados, etc all the time and these contain latex, and it doesn't cause any problems. Has this happened to anyone else? /: