Are you in the mood?

My husband told me it is a complete turn off when he rubs my legs, cuddles, and/or strokes my hair and I respond with the question "are you in the mood?"
He says it is a duh moment and instead of asking the question, if I am in the mood to just respond sexually or make a move to just please him. 
Has anyone experienced this before? Is this a turn off? What do I do?
PS we only have sex around 2 times a month when I am in the mood...he stopped making moves, he says it is because I told him no too many times! Arg help?!?!