I have been seeing my f.buddy for almost a year and a half. I can't seem to break it off.

As the title states I have been seen this popular handsome guy for the past year and a half. Mind you we've had each other added on Facebook for many years, we just never talked. He is part of a local rock band. my friends and i used to go see him play. They, my friends , would tell me he looked at me a certain way I just never believed them. He's the cool hot dude and I'm well...me. at that time I was in a relationship a stable long term relationship. When my ex so and I parted ways and he found out started to "like" my pictures, see all my fb stories, and sending me msgs on messenger. After a few msgs he asked me to join him for drinks...at his place. I immediately knew what all that meant SEX. It was fine by me as that is all I ever wanted and want. He's a selfish f.buddy. not once has he reciprocated oral. And well, I can let that slide off. But I want him to touch me, I've asked him to kiss my neck and , caress me, he said and I quote " that's too intimate, way too personal" I've cut him off due to those reasons. He'll msg me and ask me to go back that . To let him show me what he can be that way. It'll sorta happen the first time I go back then the rest of the time it goes back to the usual. Thing is I can't stop and since I don't get what I crave; I keep going back to him thinking next time will be different. Want to stop cut him off. I just can't. My low self-esteem keeps me from doing so. I want to ended then I want it more. HELP