Low sex drive - Me not him

I'm quite perplexed and I'm looking for any kind of insight. Please and thank you!
I've been with my love for almost two years now, and we've been sexually active for a little over a year. But it seems like my sex drive just keeps dwindling. It was really high at the beginning, but it's gone down quite a bit, and kind of plateaued. I cannot figure out why. 
When I'm not around him, I'm much more inclined to have sex (with him, not anyone else, just in general) but then when I go over to his place to stay I can't seem to say yes. It's almost like I'm afraid, which doesn't make sense in my mind.
It's really taking a toll on me because I want to be intimate with him, but when the time comes something just won't let me say yes. Has anyone ever experienced this? Any tips or suggestions? I'm just stumped, and sad.