anti-anxiety meds.

I take anti-anxiety medication. My first child was severely injured at birth and I am really worried about what could happen next. I am currently 17 weeks pregnant with baby #2. I asked my doctor if I should stop taking it and the impact it would have on the baby. He said in about 20 percent of cases the baby goes through withdrawal after birth. But he also wanted to know how I felt off the meds. And I have a lot of panic attacks. With that in mind he thinks I should stay on it. Have any other moms been through this? I am feeling a ton of guilt.