Allergic ?

So I've noticed that when me and my boyfriend have sex with a condom sometimes my girlie starts to swell out of its flaps and everything is out if that makes sense, but I don't know if it might be because he is penetrating to hard or maybe the comdom or if I just have a weird vagina. But when we don't have a comdom on I don't think it's ever done this. What do you think ?