Condom problem!!

So me and my boyfriend have been using condoms because I'm not on BC at the moment. But there's been a couple times he's continued after he's finished without taking the condom off?? And I think one time it busted. I'm suppose to start any minute now, should I be worried or??