Birth control effecting fertility

Brianna • Hello! I`m medical assistant, book nerd, movie buff, collector of purses. My hubby and I are TTC our first child!
I was on birth control since I was 15, to regulate my cycle and for migraines,I'm now 23 and have been struggling to get pregnant. I was told because I was on birth control for so long it may take me lifer to get pregnant. Is there any truth to this? Should I be worried?