why do condoms hate me?

So, I'm in a new relationship and my guy and I have yet to have sex. I have the paragard iud and have not had sex since it was inserted. sadly, every time I have sex while using a condom, I always end up with a yeast infection. I really want my new guy but dont know what to do. condom=yeast infection or iud=pregnancy paranoia.