Guy problems

Okay so I've been talking to this guy lately and last night he, a group of our friends, and I went to the movies to see the new Rogue movie (which i hated) but anyway I had my legs in his lap and it was fine. But then he started playing with the hole in my jeans and like tickling my knee mindlessly. I guess I would've liked it more if I wasn't so conflicted about this guy, but I literally had a panic attack because he touched me. And it wasn't like sexual or anything but it was just the fact that he was touching me. He's super sweet and nice so why don't I like him? Why did I freak out??