Go off birth control for good?

I am 30. We have 3 kids, and our last one was a surprise, but I wasn't on birth control, and used natural family planning method. It worked for years for us, but I ovulated early that one month and now we have 3 🙃

My reason for going off, Is solely on the fact that I want to better my health. I suffer from anxiety and panic attacks, and birth control seems to make them worse. When I was off BC I could manage, but hardly had any. I HATE condoms, but that would have to be our source of protection. Neither hubby or I want to get fixed. But I cannot do 4 kids 😂

Did you feel condoms put a damper on your sex life? Or didn't change it at all?

Hubby is fine and says he would rather I not be on BC pills.