I'm freaking out, please help :(

My boyfriend and i had unprotected sex on Monday and although I'm on the birth control pill I took plan B the next morning to be safe(Tuesday). I was supposed to start my period Wednesday but it's Friday and I still haven't started yet. Should I be worried?? Cause I'm freaking out right now. Please help!!!