Gave someone my number

I really regret giving this guy my phone number. I just got out of a relationship but we are still living together. Me and my ex have been up and down our entire relationship. Very rocky, a lot of damage, and eventually toxic. We both love each other and are being nice to each other at home. I feel so guilty for giving this guy my phone number. He was cute, we joked around, and I just felt flattered because my ex did not make me feel wanted. He didn’t compliment me, wasn’t very affectionate, didn’t like kissing, any pda, wasn’t really into flirting or foreplay he kind of just went straight to touching me and it was annoying. I felt used and even after telling him this it didn’t change. Anyway, that’s part of the reason we aren’t together. I’m not interested in this guy at all because I’m just not in that place. Again, I just feel so guilty, almost like I cheated. Me and my ex are not in a relationship and have both agreed we are toxic together and he doesn’t want to do the things I need to continue the relationship (therapy, better communication, and to work on stop drinking as he struggles with alcohol use disorder). Why do I feel like this though? It’s not like I’m going to form a friendship with this guy so why does it matter? He already text and asked if I wanted to hangout with him and his friends (at a bar then go line dancing) but I said I was going to stay home. Next time he texts should I just say I’m getting out of something and am not ready to go out but thanks for the invite? Also is what I did wrong? Like am I messed up for that?