My fuck buddy is an idiot

Ok so i just read that anything that enters the vagina with semen on it can impregnate you, i realllly dont think the guy i just fucked would know that, hes kind of an idiot. Anyways, yesterday we had sex and the condom broke. I was facing the other way so i didnt see, but i swear i felt him cum on me, which would be no big deal except then he proceeded to finger me for a while. If his hands had semen on them, should i be worried?