Is this safe?

Me and my bf have been thinking about having sex without a condom. I am on the pill and he said he would pull out before he came I’m just really paranoid. How safe is it?