Pull Out Method makes me think im infertile

So, my fiancé and I wouldn't mind having a baby so we use pull out method as birth control. He doesnt use the restroom in between, sometimes we still use pull out on my most fertile days. Its been maybe 2 and a half years now using pull out and there hasnt even been a scare. Should I be worried ?