annoying but cute?

There is this guy in my class. We have known eachother for about five years now. 
We have been friends for teo years. Half a year ago, we kissed at a festival. At a party two weeks after that again. After that nothing "happened" we hung out after school a couple of times. 
But heres my problem: he is this class clown type of guy. Not what I'm usually attracted to. In school he annoys me a lot. But when we hang out he is nice and cute. He likes me, he told me that. But I don't think that I like to be together with him.