unprotected sex

I've recently gotten together with an ex boyfriend of mine and we aren't officially dating yet but I've gone over to his house to spend the night about 5-6 times in the last 2 weeks and we always have unprotected sex but he always pulls out because I'm too scared of him coming inside of me. Well a couple of days ago he did come inside of me and I'm freaking out, I take the pill but the times that I take it are irregular (usually between 11am and 3pm) and I'm not sure what to do 😪