Forgot birth control one day

So I am usually very safe with my birth control and I've been on it for about a year now, always take it around 7:30 in the morning. I do not take the placebo white pills, but I do stop my birth control when those start in order to get my period, then start up again after it's over. I did not take the pill on Saturday Morning and got my period Monday morning, it officially ended this last Saturday. I forgot to take my pill that Sunday after my period was done and had unprotected sex with my boyfriend (he is usually always on time with the withdrawal method, but we aren't sure if he pulled out in time in this occasion) since I missed that one pill should I be worried? I also took two birth control pills after the unprotected sex which I know have no effect but as a nervous person I prefered taking them anyway.