Advice? Could I be?

Hello everyone 
I have a very irregular period. I had sex 4 days ago and took a plan b pill yesterday (3 day since sex). He also said he was sure he didn't cum in me but bought the pill for me just in case. I'm suppose to start my period in 5 days. But like I said its irregular. Sometimes it comes 2 weeks later or so. Should I be worried if I don't start the day predicted or a week later?