Unprotected sex

I had unprotected sex for the first time, but we stopped WAY before he came because we didn't want to risk anything, I took a plan B that same day, then 4 days later we had protected sex but I'm on my fertile window.. Should I be freaking or no?