Late period

My period is three days late last time it was late I wasn't pregnant. Should I be worried this time?