Text Decipher Needed!

So I've been friends with this guy for a long time and have enjoyed our friendship. He's never made moves on me neither I on him. We flirt occasionally as but never turned it into something serious. I need some quick advice on a text he sent and wanted to know what people thought:
We were talking about him babysitting and how the kid was being terrible. I was messing with him since he's a grudge holder sometimes and said, "It's not fun when the grudge is held against you is it?" He said, "This is no fun...". I replied with, "Oh, poor baby" followed with his reply, "I hate you" (in a joking manner of course). I said, "Nah you love me." This is the response I need help with: he then said, "Hmmm I wonder to think, that, that is how things are." What are your thoughts and opinions on what that means? Am I reading too much into it? I do enjoy our friendship and don't want it ruined by assuming too much out of something so little if it is.