On the pill & don’t want to use condoms

I’m on the pill and my SO and I ran out of condoms.

We went 4 years if not longer of having sex without condoms or birth control all he did was pulled out. We didn’t get pregnant until we tried to conceive. I should be fine to go ahead without condoms right since I’ve been taking pill regularly? Am I just being paranoid?

I am on the mini pill - Norethindrone