I'm a little worried

I had sex the other night, three times I think, and we had sex with a condom the first time and he came inside the condom. then the second and third time were without one only he didn't come because of interruptions, so I thought I was safe, but then my friend started worrying me saying that if he put his penis back inside me without a condom there could be leftover sperm that could get me pregnant, even if he didn't come. I'm just wondering if that's true?