Sex with a condom

Me my partner and I have had sex without a condom and we thought we would be safe and use a condom but it's difficult to get his penis in me with the Condom, but is easy without the condom, why is this ?