Bipolar and Pregnant question

Hi! I just found out I'm pregnant and am on medication that I have to stop. My Psychiatrist told me to just stop taking them and when I expressed concern about just stopping for fear of a miscarriage he said I could tapper but the first trimester is crucial and he didn't tell me how to tapper! Does anyone know if just stopping medication is ok?! I'm only 3w4d but I'm so scared I'm going to hurt my baby already!!!