Sex during placebo week?

My boyfriend and I don't usually use condoms, only during the week of my sugar pills because he's worried about me getting pregnant. We use the withdraw method as well, and I've tried explaining to him that since I take my pill correctly, having sex during that week poses the same risk as every other week of the month. However, his hesitation makes me nervous so I'm just wondering if what I think is correct?