Bc pills messed up my sex drive???

So about 2.5 years ago I was on bc patch didn't get on with it as I feel it took my sex drive away so I went on to the pill.. I've not had a sex drive since so I decided to get some bloods taken to see what's going on.. So I called the doctors up an thay say everything is fine with me so I then say to my husband could all this time been the bc pills !!!! I'm starting to think it's down to the pill that I've got no sex drive. So I'm thinking of coming off them to see if I start to get it back again but I can't use condoms as I get a rash an it's not nice put it that way but I don't won't to get pregnant???