Fiancé always turns down sex

We've been together for about 3 years. Just this last year, he has been constantly turning down my advances towards him. I don't feel sexy anymore and when I try to talk to him about it he makes me feel bad for even thinking about it and/or it starts a fight so I am left sad, mad, and horny. We have sex once every week or so, but it's never that in-the-moment sex. And it's usually provoked by him wanting it. We aren't prudes at all, but he acts so weird sometimes. Has anyone else been with a man like this? What should I do? I'm going crazy!