Quitting birth control & TTC

Amy

I've been on birth control pills for about 3 years straight & two weeks ago my husband & I decided we wanted to TTC. Is it true that it takes your body a few months to adjust being off of the birth control and it makes it hard to get pregnant soon after you quit or is that just a rumor?