question about condom use...

I'm 17 and my boyfriend and I have sex fairly often, I have a paragard IUD and we always used condoms too. But he has a lot of trouble reaching orgasm, and had never been able to in his life until last week when we tried having sex without a condom. Today we had sex again, and he pulled out both times, but I'm still nervous and I told him I would rather use condoms as well as my IUD. He is fine with this, but does anyone think that we took a very big risk? I'm just nervous.