Long time friend & love interest

I've been friends with this guy since 2008. We have always flirted, but nothing ever came of it but it obvious we have had chemisty. During this time we both dated other people. Flash forward 2012 we both get drunk and make-out. Nothing comes from it and he gets a new girlfriend. Flash forward this New Years he ask to come to my city. Which is two hours away from where he lives and spend New Year's with me. We kiss we hold hands. He asked me on official date we talk & stuff. He comes down again to visit with his friend he makes one hour to spend with me and nothing really ever comes out of anything. Then I decide to visit him where he lives two hours away when I get to his apartment which I've never been to before he has pictures of him and his ex-girlfriend hanging in the living room. Then he goes on to say that he forgot I was coming that day. He didn't come up with anything special for us to do or any fun activities we did have fun that weekend but I assume that he would of planned something or attempted to make the weekend special since I traveled a long ways to see him. He seemed to really hot and cold sometimes he wanted to be lovey-dovey and sometimes he seemed very standoffish like he had a wall up or something. He never calls me I always have to call him. And when I do call him he seemed very uninterested. He only text and sometimes it takes them forever to respond. I'm not sure if he's still in love with his ex but I'm getting the feeling that maybe he's not interested it's just that we've been friends a long he doesn't know how to break it to me. I need help deciding should I stop talking to him or should I back off a little bit and allow him the chase me and if he doesn't chase me then he's not interested and leave it alone. PS we have never had sex. Help !!!