Is changing birth control methods bad for you?

First I was on the mini pill but it made me bleed after sex (and have a low sex drive)

Then I changed to the combination pill but my sex drive was still low and my periods were quite painful.

Now I’m on the evera patch and my sex drive is still low and I’ve also found a lot of downsides to having the patch.

My nurse told me that low libido is caused by the progesterone in the BC (and I’ve only ever had ones with this ingredient) so I want to change to a pill without it. I don’t know if it is bad for me to change another time but I just want to find something that works for me.