Does your birth control affect your anxiety?

I don't have depression, but do have pretty severe anxiety with panic attacks triggered by random but predictable things that don't make a lot of sense... I'm not on any meds anymore because I didn't like how they made me feel, and I was doing ok. Until I started taking the pill last summer. It was also around that time that some other stressful things were going on in my life, but I couldn't help but notice that I started having way more panic attacks as soon as I started on the pill. Just as an experiment, I decided to try stop taking the pill for a couple months, and I did notice the attacks becoming less frequent. Has anyone else had this experience??? I need to be on birth control, but I'm so afraid of my brain getting out of control again :/