a little worried

I'm on the pill but I took my pill 4 hours late today and I had unprotected sex with my SO earlier in the day and he finished inside me. Should I be worried??