Pregnancy and the pill

So my husband and I are finally ready to start trying to conceive! I'm hearing horror stories of people being on the pill, stop taking it and having issues getting pregnant because of it. Is this true? Does past use of the pill cause fertility issues??