Am I being unreasonable?

Everyone in our classes say we are a couple etc. And it bugs me when people always say we are perfect for eachother because he gets on my nerves, and I only see him as a friend.

Today we were walking into college with his friend Meg, and she noticed me and him were wearing almost matching tops and similar jackets. WE WERE MATCHING. Que the "aww". Immediately I grabbed a jumper out of my bag and pulled it over my head, and he got all offended that I didn't want to spend all day trying to tentitively explain to people why we aren't like that. He wanted to stop and talk about it bus I just wanted the problem to be let go so I could go to class.

Am I overreacting? He says our tops aren't even similar, but then he also don't see everyone staring at us when we were arguing. He seems hurt by it, but why is it his feelings should take priority over my frustration? I don't know what to do..