Low sex drive ruining our relationship? :/

My boyfriend and I have been together almost 4 years now. In the beginning, I'd say our sex life was very active. We'd go at it everytime we saw each other. But a little later into the relationship I went on the pill, which drastically changed my libido. We still have sex, just not nearly as much as we used to. We've had talks about it, and pretty much every time we did it would end in me crying because I feel guilty about it. I know he wants sex but most times I'm just not in the mood. He would never get angry or anything, but there have been times where he would try to initiate and I'd shoot him down, and he'd get really quiet and sulk. I'd feel extremely shitty for this everytime. I would try to talk to him about it but he would say that he's fine and roll away from me.

I constantly feel guilty and scared that this will end up ruining our relationship. I hate that I make him feel neglected, but I don't want to end up forcing myself to get in the mood when I'm clearly not. I know things aren't the same, and I don't know how to get that back. I've gotten very reliant on the pill these past few years, so I don't see myself stopping that. How can I go about this?