Sex without a condom

My partner and I have only ever been intimate with each other. I've been on the pill for a long time, and we use condoms too. We recently tried for a few seconds without a condom and I found it significantly less painful and more pleasurable. We're considering ditching the condoms but are worried about pregnancy. Thoughts?