Sperm outside of condom?

Earlier I gave my boyfriend oral and he ejaculated in my mouth, i swallowed it and kinda kept going for a minute afterwards, then we had protected sex afterwards. He told me the first time he put the condom on he had to take it off and flip it inside out so it could fit right. I’m just now finding this out; now i’m afraid there may have been semen residue on the condom from when he flipped it inside out.. even though i swallowed & it could’ve just been spit. Should I be nervous about this?