On the pill. Came inside

Just had sex and he came inside me. I’m on the pill but I sometimes take it a few hours late (including tonight) Should I be worried?