Did something stupid

Had sex with my BF of 4+ years last night. The first time we used a condom, second time however, I had a bit of wine and since I'm about to leave state for the next 6months I kinda jumped him. He asked if I wanted a condom but I was a bit tipsy and kissed his mouth shut. He came the first time that we had sex but when we weren't using a condom he had like a midpoint ejaculation. It wasn't a full finish and he thinks he pulled out but I am working a huge job starting next week and I can't afford to get pregnant. We then had sex two more times, (after I took a shower and peed). Should I be worried or take a plan B? I know it sounds silly and I have been educated however, my sister got pregnant at 19 because she was one of the 0.01% that a properly applied condom didn't work for so I'm a little nervous. We always are protected which is why I was so dumb. (I'm also on birth control but I take it around the same hour everyday but I'm not super consistent cause I work a weird job and they won't let me carry my pills.)