Should I be worried?

Me and my boyfriend have been having a rough time. I am on the pill but sometimes don't get to take it at the very same time each night. We were fooling around and we didn't use anything else and "do it" long and  he didn't orgasm in me. Should I be worried?