Do I need a plan b?

My boyfriend and I had sex twice today. First time was no issues, he pulled out and finished in the condom. He grabbed a towel and wiped himself clean and dry. Then about 10 minutes later he was ready to go again. The second time he didn't finish at all, he pulled out and we noticed the condom wasn't there, I felt myself and felt the rim of the condom. I reached and pulled it out. It wasn't in me all the way, the rim was outside of me. Do you think I need a plan b? Again, he didn't finished the second time..but could he still have had sperm on his junk? I dont know, freaking out a little. I'm not on any birth control by the way. Please give me advice.