How do you break up with a guy you're not In a relationship with??!

So about a month ago this guy and I started talking , I had been knowing him for a while through mutual friends but we never talked like that, and things were going Great and I was really into him but a situation happened and we stopped talking in that time we had stopped talking my ex and I actually got in touch and have been inseparable since he was my last boyfriend and we only broke up bc I had a disagreement with his family and we had decided to take time apart so when we got in touch things were amazing we've been going on night time adventures (that was our thing) and just enjoying eachother and just as things were getting back to how they used to the other guy comes back into my life and starts talking about how he misses me and I was trying to be nice to him and tried to pull away but he didn't stop and the other night we decided to hangout we were drinking a little and he kissed me and one thing let to another and at the end of it he tries to say that I said I loved him but I wasn't drunk so I know forsure I never said that . But now he's talking about having kids and being together and how he loves me and how he's never going to leave me even if I try to and I just don't know how to end things like I really care for him but my ex and I are trying to work things out and I don't want to ruin things bc of this guy how do I let him down easily bc after all he is a good guy and tbh he's one of those sensitive guys I've never had this happen to me I'm a one man girl before I met this guy I had only ever been with my ex sexually and he is my longest and most serious relationship so I don't know how to approach this situation

Please help !!

Side note: my ex and I aren't anything yet we're just hanging out.