Am I wrong?

I drove to this guy who lives an hour away once for our first date. He told me he wants to do a redo of our date because it should've went better. I only have $10 to my name, I start working soon though, but I'm broke. I can't drive an hour to him and back like that. So I said that he could Uber it (he works and gets paid well) or get a car, and he said no. So I said "oh well then. If you really wanted to see me, you'd find a way" and left it at that. I guess we won't be seeing each other at all then.

I don't think it's fair for me to constantly break myself and find an effort to see him without his efforts being reciprocated. Was I right to tell him that? Even when I start working and would be able to drive to and from where he is without the worry of gas, I still think it wouldn't be fair.