Birth control pill + pull out method ...

I have been on the pill for a little over a year now and I take it at the same time every day. Every time I have sex with my boyfriend we use condoms EVERY single time because I’m so worried about getting pregnant. About 2 weeks ago, the condom broke and he came inside me. I started my period and took a test and it was negative so it really eased my mind because I was convinced that I was pregnant. What I’m trying to say is, if I start using NO condoms and using the pull out method only, is there a greater chance of me getting pregnant on the pill? Should I continue to use condoms?