My man doesn't want to wear condoms and I'm scared it will ruin our sex life!

When I met my man he was a virgin. He had trouble getting it up Bc he was so nervous. The first time he could get really hard we didn't use a condom, and ever since then whenever I'd make him wear a condom he A.) couldn't stay hard B.) would NEVER finish. We also used a condom and it broke twice!!! It became really frustrating for me, so I eventually caved and let him go without a condom. I take the pill every day and he pulls out, but every month without fail I have anxiety attacks about being pregnant. I want him to wear condoms again but I'm scared it'll go back to him not being able to feel anything, especially having known the raw dog experience for so longšŸ˜© ladies what do I do!?