Anti anxiety med question.

So a little over a week ago I was told to go to the hospital because I passed out. Baby girl is fine and after a long, exhausting 24 hours of some pretty severe nausea and dizziness I was sent home. The doctor I saw said that it could be a combination of the medication I was put on when I was around 14 weeks pregnant. I currently take Zoloft and Buspar. He suggested I try coming off the Zoloft. I have been taking half a dose of the Zoloft and come Tuesday (2 weeks since I started half a dose) I'll go to half a dose every other day. Is this safe for baby? I feel fine, but I just want to make sure I'm not doing this too quickly.