Anxiety/Panic attacks quitting birth control

Jolijn
I had been on the pill for about 7 years until I decided to stop because I felt like it was time to get off the 'fake hormones' and try something new.
However, since I stopped I have been experiencing panic attacks since the first time in my life, and though I am going through a rough time right now, it was been very scary to have anxiety all of a sudden.
I have been seeing a psychiatrist who gave me some Xanax to sleep the few days I had constant panic and are slowly feeling much better as though it didn't happen ( I had 3 major attacks within 3 weeks). 
Now I know I should probably visit the psychologist to talk about my anxiety, but I am a bit scared to do so. 
Anyone aho experienced the same? Or any tips?
Thank you :)