Accidentally used latex condom, and have latex allergy. Help!!!

I'm really allergic to latex. I once forgot to tell a doctor before an endoscopy, and ended up with lips the size of a blow up doll, and blisters all up and down my throat and the inside of my mouth, a fever, and unable to eat due to swelling for over a week, just to give you an idea of how bad it is. Due to this I've never even tried using a latex condom. Last night the guy I'm dating accidentally grabbed the wrong condom. It was dark, and he thought he grabbed the non latex ones. Neither of us realized this happened until this morning. Normally I'm a light sleeper, but I completely knocked out after we had sex. He was surprised because he said I didn't wake up at all during the night like I normally do, and I felt clammy and cold. I woke up in the morning and have been having stomach pain like cramps only on the left side, and a strange inner thigh muscle pain. My vagina seems like it's looking pretty normal though, maybe (big maybe) just a little redder and a tiny bit swollen. He thinks I might be freaking out for nothing, and that I may just be feeling those symptoms because I knew it was latex, if I didn't maybe I would be fine now. Which is possible, I didn't notice most of those symptoms until him and I saw the wrapper in the morning and realized what happened. But I also know with me it can take hours or even almost a day after exposure for symptoms of the latex allergy to pop up. Am I being overly paranoid? Should I wait and see what happens? I just bought benadryl and took 2 tablets. anyone else with a pretty severe latex allergy ever use a latex condom? What ended up happening?