Would you be upset if your husband said this?

Husband and I were having a joking conversation tonight and he said he was going to pick up dinner but might be a bit later because he was seeing a friend real quick too. I made a joke saying meeting your mistress? Lol and he said "if only you would let me". I said wtf? And he said "if you told me I was allowed to have a mistress I probably would". Dead faced, not joking at all. Would you be upset about this?

EDIT: He's mad that I'm upset about it! Apparently he said it "sarcastically" because I accused him of cheating😑 Such a crap excuse. He just stormed off mad at ME for being mad at him.