Sex on Birth Control

So I'm on BC and my boyfriend and I pretty much never use a condom. When it's time to take the last 4 pills at the end, I don't always come on my period for all 4 days, so is it still okay to have sex without a condom during those days? We haven't yet, but he asked about it and I realized that I don't actually know.