Nervous about sex without a condom

So my boyfriend and I have been together for a while and always had sex with a condom (SAFE SEX FTW KIDS) even though I've had the skyla IUD for over a year now. I know it's slightly irrational fear but I'm terrified to have sex without the condom. Does anyone else have this worry? All my friends keep talking about how great the sex is without the condom but I'm just too scared to try it.