Sex during last week of pills

Can you still have sex without a condom during the week of inactive pills? I already had sex without a condom and I don’t know if I should be worried about pregnancy or not.