Condom dilemma

I'm allergic to condoms. Well, I'm allergic to the lube, or some lubes. I don't even know 🙄. All I know is when I have sex with a guy who's wearing a condom, afterwards I HURT. My vagina gets inflamed, it feels like it's so swollen that it's rubbing against itself inside and it gets all dry. This even happens now when I use tampons and pads, so I've had to switch to a menstrual cup and cloth pads. 
Anyway, I haven't been sexually active for a while so I wasn't too concerned about the condom thing. But I've started dating someone again and I'd like to figure out what may be causing this. I'm definitely not allergic to latex, and I know some condoms don't hurt nearly as much as others, but I don't know what the common denominator is.
Does anyone else struggle with this? What condoms do you use? Does anyone know of hypoallergenic condoms I can use or something???