Saying no to sex....

One of the biggest struggles I have in my marriage is how much lower my sex drive is than my husband's. We have talked about it before and I have explained how I need to feel like I can no because in the past when I have he gets very hurt, offended, and angry. How do you handle saying no to sex? If I end up doing it when I want to say no sometimes I feel used and dirty and makes me want it even less next time. I know my husband loves me and doesn't want me to feel that way but he still gets upset when I turn down sex :( I would love to hear how other people deal with this kind of situation