Latex allergy?

So a few weeks ago I had a one night stand and we used a condom... But when I woke up the next morning, it hurt a lot down there and it was red and a little bit swollen. I've had sex without a condom only once in my life and I don't remember feeling that way the next day. I've recently been tested and I don't have any STIs and my mom and grandma have a history of latex allergies. So pretty safe to say I'm allergic to latex, right?? Thoughts on latex free condoms?