So stressed now about my anxiety medication

Ok so my nurse called to do the 30 min history on me before my first appointment next week. I'll be 7 weeks tomorrow. I have horrible terrible anxiety and panic attacks. I've been on xanax for 2 years. .05 so a low dose . as soon as I told her this she seemed very concerned and said that is a red flag and can cause birth defects and is not safe. So I have to call my doctor that prescribes it to me. But ive read online that alot of women still take it during pregnancy and there's a 10% chance of birth defects. Alot of them said the benefits out way the risks per their doctor, and there babies where born fine. With no problems. I've read its a horrible withdrawal. And I don't know how I'm going to live with the panic attacks of I have to go off of it completely. My baby's organs have already developed so I'm really mad they didn't ask me about medications when I made my appointment 4 weeks ago when I found out. So I can't just stop taking it until I talk to my doctor. And I'm just freaking stressed now