Condoms reactions !

Is it me or does anyone have this problem latex condoms irritated me and make my vagina swell then gives me a yeast infection . It's so annoying is there anyway I can prevent this ?