Late pill

I take my bc combo pill perfectly expect for one day I took it four hours late. My bf and I had sex after I took the pill late and he withdrew. Am I protected or should i be nervous.