Birth control for a long time

So I've been taking birth control for about 10 years now. For the first 5 years I was on the pill and then I switched to implanon for 3 years and I've been on the pill again for about a year in a half. My husband and I are not TTC just yet because we want to wait until we build our house next year. But I'm wondering if we should at least start trying because I've heard if you've been on birth control for a long time it may be difficult to conceive. Has anyone had trouble conceiving after being on birth control for a long time?