Implant removal - pregnant straight away?

So I got my implant removed today as it was getting bad periods from it. I explained to the doctor that I wanted to give my body a break and just use condoms for a while. He asked when I last had sex and I told him yesterday, he said that I needed to go on the pill straight away if I didn’t want to get pregnant from the sex I had yesterday. My understanding is that it can take a while for your body to ovulate again after getting the implant removed, and that the pill takes up to 7 days to become effective meaning that I wouldn’t be covered for the next week anyway? I really don’t want to go on the pill, but I’m worried now that it’s possible to become pregnant straight away?