SO cum or mine?

This has happened quite often. 
When me and my SO have sex with a condom, his sperm inside, and mine outside, how can you tell which is yours or which is his? 
Because I keep freaking out whether his spilled out of the base of the condom.