Sex with my ex

I broke up with my ex 4 years ago and we had a season of off and on again for a couple of years. But I eventually got fed up with him and blocked him. Turns out he made accounts to reach out to me regardless and I let him back into my life. He’s like a drug. The other night he came over and we had sex and I know better and I feel so guilty, but I miss feeling cared about. The sex was the best sex we’ve ever had together (it has never been amazing, but this was way better than normal). After we finished he was playing with my hair and smiling down at me, but then as soon as I went to cuddle he got up and made a bunch of excuses to get out of there and leave. I know he wants nothing more than sex and he’s a terrible person, but every part of me wants to unblock him again and tell him to come over tonight and have sex with me again. Am I the only one that has that one person they can’t stay away from? How do I stop? How do I get him to stop reaching out to me?