backup?

My boyfriend was inside me without a condom for no more than a minute. I'm on birth control but missed a pill on the 21st of October and my pamphlet said to just take the one I forgot when I remembered right away and then one at my normal time and that I shouldn't worry about other protection. I've been taking it regularly for almost 3 months and that was the only day I missed. I had a period for about a week because of my missed pill and I'm on my placebo pills now and I'm bleeding a little bit of brown blood so I might start again. Should I take plan b? Should I worry?