Back on BC

Soooo my boyfriend and I decided we were going to TTC this month so I didn’t take my birth control pills for 5 days. Then, I talked to my doctor and she told me I would probably have to taper of the medications I am on before trying for the risk of early delivery and such. We had sex on day 4 two days before my fertile window. I took two pills today and will continue to take 2 a day until I’m caught up. Should I be fine? Or could I have got pregnant two days before my fertile window?