Question to all you ladies...

Can a guy get mad at you for not wanting to do stuff if u made a joke about it?

Example: So me, my friend(girl) and this guy went to eat dinner together. The guy made what I thought to be a joke about me "repaying" him for him teaching me math one night. And I was like "haha yeah ill repay u" but I said it jokingly. He also knows I don't wanna do stuff cause I have declined many times before. He took it seriously and was trying to drive me to his room after and I said no I was just joking and then he got upset. I don't think that is justified but what do u guys think?

P.S we have previously done stuff but I made it clear to him like a month ago I don't want to anymore.