is there something wrong with me?

About 2 years ago, I went out with my cousin to a friend of hers birthday, I wasn't really having much fun and then I met his brother.. it was just the strongest connection when we met... we both could not stop smiling with eachother, so before I left we exchanged numbers. That night I texted him something super sweet and cute and he responded to me how sweet I was and he loved meeting me etc. after that we had started talking a little more... only thing is, I came to find out he was 45 and I'm 24.. but honestly I didn't care about that, and he didn't seem to mind. We hung out maybe a total of 6 times within this 2 years, and seen eachother on multiple occasions because of his brothers get togethers and anytime we are around eachother, he just can't keep his eyes off me and is always all smiles. He has never tried to just have sex with me or anything, we have kissed and he's fingered me, went down on me and that's it. It's starting to get harder for me because I'm really into him, and I think he's really into me but I think he might be scared or something, he always says "I'm trying to be your friend" but then he says "you're so beautiful I love being around you and talking to you" and occasionally when he sees me he just touches me... but always says he's trying to be my friend but when I try to keep it friendly he wants to kind of spark something up... I can barely get him to have any real time but I honestly don't think he has someone else... he's very open and upfront. I just don't understand what to feel or do...