When should I to be worried I’m pregnant?

My period is 4 days late, I know that’s not that bad. When should I start to get concerned?