TTC & medications need advise!

Has any of you women got scared you wouldent be able to conceive because you took medications for things such as high blood pressure and depression only to stop taking them without telling your doctor because you thought it would make your chances higher to conceive only to find out it got harder to conceive because you stopped your meds??? I feel dumb but need some advise!! Please