Help!!!

Me and my partner had sex earlier and then had sex again, but when i was trying to put on the new condom i put it in backwards and he just flipped it around and used it like that...does it mean the condom had sperm from earlier cause it touched his penis and we had sex?