birth control

So I'm on birth control, I can't remember to take my pill at the same time everyday & my boyfriend basically never pulls out. I have been with him for about 4 years & I don't know if I should be concerned because I skipped a pill about 2 days ago & the last time we had sex was 3 days ago.