TTC on Zoloft

I conceived my dd within a month of coming off birth control 5 years ago. Shortly after she was born I started taking Zoloft for anxiety and have been on it ever since. We’ve been trying for number 2 for almost two and a half years now and nothing. Had fertility tests and nothing apparent wrong with either of us. I’ve decided to wean myself off Zoloft slowly as I can’t help wonder it’s stopping us conceiving. Has anyone else had this?