His way or no way

My husband has been irritating me for like the past year now. He told me that he feels like Im not attracted to him because I never come to him for sex. He always has to come to me. He's not lying though. But I never come to him for it anymore because he turns me down literally every single time. Then as soon as I stop coming to him, he starts complaining about it again. I feel like he's playing some type of game with my mind! I explained my reason for not coming to him and he said he doesnt know why he does that and he apologized to me. Even though it hurts my feelings when he does that, I never get mad about it. But if I turn him down even once because im not in the mood I have to deal with his attitude and smart comments all day. Idk what to do anymore. Everything else in our marriage is great. Any advice on how to solve this problem? Or if you were in this situation, what did you do?