plan b and the pill

so I've been taking the pill for over a month and half and lost my virginity in the am on sunday, i take my pill everyday within an hour of 2pm and haven't missed any. whilst everything moved super quick we didn't use a condom!! Tuesday i went to the doctors and he said i shouldn't worry about being pregnant since im on the pill and use it correctly but i took plan b anyway! should i be fine from pregnancy?