Are condoms really THAT bad?😂

So my body is really too messed up at the moment for me to be on a pill, so my fiancé and I resort to condoms. But the other day, he said something that kind of disappointed me:(
" I can feel the love so much more when we don't use condoms. I can feel you so much better and it feels so more natural, I can feel that you trust me more when we don't have one, like we have a better bond." 
I feel so aweful because now if we have sex with a condom, I know he won't like it as much, but without a condom and I'll have a horrible fear and anxiety of being pregnant for weeks, and that's assuming I wouldn't be pregnant, lol. 
It just really sucks knowing that the only way for me to have anxiety free sex is if he doesn't enjoy it that much ;( idk, I just wanted to share feelings with you guys, sucks to be a girl trying to protect herself😔