Painful condom sex!

autumn
So when I have sex, bare skin, it feels great. feels like it fits perfect and everything. but everytime I've used a condom, when the head goes in, it starts hurting really bad. like stinging. it almost feels like there is a tighter hole inside of the original hole that the penis is ripping inside of. I hear the latex allergy thing, BUT wouldn't other things happen if I was allergic. I know it's not just different sizes because it's the same guy and when there's no condom, it's great. but when there is, it stings and feels like he's ripping the inside of my vagina. help please??