too soon to give it up?

I've been talking to this guy that I really like and we are going on our second date tomorrow. I haven't had sex in a very long time, and when we went on our first date the was a lot of sexual tension between us... he has already been to my house so I suggested that after our date tomorrow we can head back to my house to watch a movie and cuddle. Which he was down for at first but not even a minute later he texts me back saying it might not be a good idea since it might lead to more... I told him I did want to have sex with him, but I was scared that he would just get sex and the next day just drop off the face of the earth and never talk to me again. He asked me why I would ever think he would do that, and I told him that guys have done that to me in the past... he told me that it was okay and he was fine not having sexy tomorrow... but I don't know if I should give into my feelings and just do it and forget about the possibility he might not call me back in the morning, or wait until at least another couple of dates to know he is going to stick around? I don't know what to do and would really like some advice please