Sex with someone in a relationship?

I know that I'm probably going to be bashed for this but I honestly need advice so don't be rude. Anyways I have a guy I used to talk to in high school but we fell off and we talk on and off. So we were talking about being sex buddies and I asked him like do you have a girl or anyone that claims you and he's like slick. So I'm like what is slick and he's like it's not that serious but here's the thing I do want sex with him but I'm wondering if I should just tell him straight up like I can't do this. He thinks we are meeting up tonight. He just texted me asking for.my address. I haven't responded. Please help.