no condoms

me and my boyfriend had sex today a couple times and the first time was without a condom and the other times we used a condom. i am on the pill and have been for about 2 years now. should i be worried?