I want to have Sex w my Ex

I'm a Virgin and my ex isn't. I still talk to him on a regular basis but I've found that I can only get horny when fantasizing about him... We messed around a lot when we were together but never got past feeling each other up. I desire kissing him and I feel like I'll never find anyone I'll want to just kiss that much. The sexual chemistry is strong basically. The problem is, I've told him I want to & he does to but I know it's a bad idea. I want to be alone with him and we've been trying to make it happen but I know I'll give in and I'll regret it. I also don't want to disappoint him. It's just a huge dissonance that I can't seem to overcome. He was just about my first everything, i.e boyfriend etc.