Plan b?

So there's a guy that I had history with and I've been hooking up with again recently after a year.  Before he would not use a condom and I took multiple times plan b. After all that I did not want to take it anymore and promised to myself that I would be more careful . Now that we've been talking again he had been using a condom but a day ago, we got carried away in the moment and he didn't , but pulled out , wiped himself clean , and we ended up having sex again right away for a bit more. After we were done he had said that he had came a second time but I didn't get a chance to get off.  Later on before I left him I asked him again if he did came and he told me no.  I know the correct decision would be to take plan b but is taking it too many times bad in the long run? I really don't want to take it again because it messes up my cycles but if I have to I will.