Could it really be because of the pill?

I have had an issue in every serious relationship I have ever been in and it has caused huge problems. I have almost no sex drive. I have been on birth control pill since I was a teenager. My relationships are all pretty much the same....start out hot and heavy with high sex drive, then fizzles out to almost no sex drive. In all cases sex becomes more of a chore than pleasure. Anyway when my ex husband and I decided to try for a baby and I stopped the pill my sex drive sky-rocketed. He later accused me of pretending because I wanted to get pregnant. After my little girl was born and I had an IUD placed the same thing happened, no sex drive. BUT I  was aware that by this point I did not want to be with him and was not attracted to him at all. Fast forward to now. When my hubby and I first got together we couldn't keep our hands off of each other, had the IUD removed due to health problems, and put back on the pill. Then he and I started having the same problem. Sex felt like a chore. I LOVE my hubby and am attracted to him but I just didn't EVER get "in the mood" sometimes with foreplay I would but rarely. And we all know men love for their women to come on to them but it was just like I NEVER even thought about sex. A few weeks ago we decided to try and have a baby. I stopped the pill and my sex drive has increased. I am just worried that my ex husband was right and it is only because I want a baby. And if it is the hormones from the pill are there any other options out there to prevent pregnancy without decreasing sex drive? Sorry so long?