Condoms

So whenever I have sex with a condom it hurts like really bad when he first goes in. But once we start and actually get going it feels amazing. Once he pulls out it hurts again and is irritated and sore for the next couple days. However when a condom isn't used sex doesn't hurt at all. Could it be an allergy to latex or just from the condom itself ?