Sex isn't lining up

Husband thinks we have sex 2-3 times a week, but we don't. We often go at least a full week without sex if not more. I have to deprive myself of all sexual stimuli so that I don't get crazy horny and frustrated, but now when he wants sex...I'm just not in the mood from having been depriving myself. What can I do so we don't fight over sex? We end up mad at each other because he doesn't like hearing that I'm depriving myself of sex (he says it puts all the blame on him), but what else can I do?