5 days late.

My boyfriend and I usually have unprotected sex to start, but then we switch to protected when things pick up. then he finishes inside me with a condom on.. is this unsafe? Last month I was 4 days late (this is abnormal for me) and now this month I have been 5 days late, and still havent gotten my period. Should I be worried?