Is it harder to get pregnant after being on birth control for a long time?

I have been on birth control for 4 years now on and off. Im worried that I wont be able to get pregnant. Is it true that its harder to conceive after being on birth control so long?