I didn't have am answer for him....

I've been staying in a hotel for a bit to try to bond with my son. After he was born I wasn't ready to be a mom and left him with his dad. I hadn't seen him since the first day I left the hospital and that was 15 years ago. I paid my child support, but we never talk or I never met him. It's been hard trying to bond because every time I offer for us to do something together he would decline or say he was hanging out after school or doing after school activities. Even when we first met a couple weeks ago he was acting awkward about the whole thing. I tried to take some of you guys advice. I sat down with him and told him I'm there for him and that I love him. He then ask me "Then why did you leave?" I didn't have an answer.... I hate that I didn't have am answer. I just sat in silence for a minute. "I told him it was mistake, and that I'll always be here if he needs me." He told me he didn't need me then and doesn't need me now. He tried to walk away and I grabbed him arm. Not in a violent way. Gently so I could talk to him put he pulled it away and told me to leave him alone. My ex talked me later saying he doesn't think this is gonna work right. I spent a lot of money on plane tickets and a hotel for it not to work. I just need more time but he's saying he thinks its best if we stop trying for right now. He thinks I should just go home. I'm devastated.... What can I do to make it work? I don't want to just pack up and go home.