Are condoms drying?

This is kind of weird to ask, but I'm definitely not going to ask my mom. Do condoms dry you out down there when having sex? While having sex with my boyfriend sometimes even though I'm in the mood I'll get dry and then we can't have sex. Whenever we have had sex without a condom though it's been fine, so the condom is the only thing I can think of that causes it. Does this happen to anyone else? Are there ways I can prevent this from happening?