Husband obsessed with...

So my husband is literally obsessed with sex. He would do it all day every day if I let him. He asks me every "chance" he gets. (Like if my kids are in the next room still awake and knocking on the door every 5 minutes is a good time in his eyes) we have 3 kids. I stay home with them and take care of everything in the house, cooking every meal, cleaning up after every person (including him), all yard work, all home renovations, bills, extracurriculars, schedules, homework, wake up with baby every night with no help, etc. So when he wants me to just drop it like it's hot every time he bellows I am not one to just be like oh yeah baby do me! Am I wrong in not wanting to have sex with him every second or is he just being an asshole by being upset with me when I say no? (And we have sex twice a week at the very very least)