Am I being unfaithful or just playing with fire?

I had lunch with a guy I use to have sex with and he keeps having this idea in his head that we are going to have sex again but I'm in a relationship! I kind of went along with some of the things he said but I told him I won't be having sex with him again as long as I'm in a relationship and that I'm happy with my boyfriend. I feel bad I had that conversation with him. But I didn't have sex with him and I am keeping my distance from him. Am I doing something wrong? I don't want to be unfaithful