Having an affair what do I do?

I recently got back with my babies father, to try and see if I can love him again, however I went for a night out and slept with one of my friends, he's married ( I know so wrong) and it isn't the first time we have done it. This time he was saying loads of stuff like he always thinks of me etc, and I'm exactly the same, I just can't get him out my head! What should I do? Should I question him about it, or just leave it? I asked him if he loved her and he said no, just with her for the sake of their kids. I don't want to be a home wrecker but I can't help how I feel about him 😩 every now and then we just get attached to each other, and when he sees me with another guy he gets extremely jealous!! please can someone just give me some advice please!