how do I tell them I don't want to hangout, without being rude.

Katie
Long story short, I have this person who is only coming back into my life bc they broke up with their gf, and I don't really have time, or want the drama in my life anymore. I literally cut everyone out that I used to hangout with bc it was all negative. I have a kid now, and I can't/ won't allow that into my life anymore. 
I've already told this person I'm willing to message them on Facebook but I don't want to hangout. It's too temping to smoke again, I've been clean from tobacco and weed for almost a year now, and I really don't need any temptations. 
And it's the same old shit with this person. Idk how to tell them no again, without being rude. 
If I told you no, flat out, I didn't want to hangout would you be super hurt?