Is this annoying to anyone else?

I’ve been seeing this guy for a while and I noticed any time I ask him something that he doesn’t want to answer he says “you’ve already asked me this and I’m not answering again”. For example: today we were talking and being cute and I asked him what’s his favorite thing about me and he said “I’m not answering this we’ve already had this conversation”. I’ve never asked him that before. He does that every time it’s a question he doesn’t like and then I start over thinking it like am I crazy and just don’t remember having these conversations? And then he’s annoyed and I feel bad. It’s just annoying at this point because I truly don’t think we’ve ever discussed any of these things.