I'm on the pill but take it late sometimes

My boyfriend and I had a lot of sex because I came home from college this weekend. We didn't use condoms and I am on the pill. He was worried that he might have finished inside me a little bit. And the day before we had sex and he finished then wiped it up and then we went for another round without him washing out the extra cum inside his head. I was freaking out today and last night because my nipples are tender and because I was making myself go crazy thinking about it so I bought the Plan B pill. Should I be worried?