How quickly does sperm die?

I have two days left in my fertile window and I was giving my boyfriend a blowjob/ handjob, stopped, wiped off my hand, and then maybe 10-15 minutes later touched the condom before we started having sex. How quickly do sperm die? Should I be worried about being pregnant?