The pill and reduced sex drive

Hey, I've been on the pill for 5 years. I am have been recently considering going off the pill. I read somewhere that it was bad for your body to be on it for too long, but also that I think it might be zapping my sex drive. 
Does anyone know if it can take 5 years before it affects your sex drive? Or perhaps I just didn't notice it before while I was in newer relationships and in the 'honeymoon phase'?
I've been seeing my partner for almost 3 years and have been living together for almost as long. 
I could really use some advice here :/
Thanks girls,