Xanax while pregnant.

So I just found out I am pregnant! 5-6 weeks with baby number 2! I have a 7month old and I never had a problem like this. I never took Xanax with my first pregnancy. I was diagnosed with epilepsy from stress so I was on about 8mg a day and we were trying to get pregnant me and my hubs and so I started to taper off! Now I have fertility problems due to PCOS so I'm on 1mg a day and I asked my dr n he said it was ok to stop right away but another dr told me to taper off so it doesn't shock the baby and I don't lose my pregnancy and it's not my first miscarriage so I'm worried. I'm glad it's only 1mg. Can I just stop taking it? What will happen? Has this happened to anyone else? I don't want to take it if I don't have to, but will I withdraw? Just advice and stories needed not any negative stuff here please I won't answer it.