Hooking Up

This guy and I have been hooking up (we've done oral repeatedly, but have never really moved on since there). Last night I told him I would be willing to/wanted to have sex with him if he wanted to, and he said he wasn't sure. After we made out for a while he said sure, and we started having sex. After probably like a minute or two he pulled out and said he only felt comfortable going all the way with people he was "with." I've told him I had feeling for him, and he said he cared, but we've never gone further than late night, usually drunk hook ups. He seems nice enough in the moment, but doesn't seem like he truly cares after he's sobered up. After a year of this, should I just drop it, or tell him I need more?