As a concierge would it be weird to be getting food/dinner with a resident

I work as a concierge but a resident asked me to get food and I’m a shy person. He seems normal and hasn’t hit on me. He knew I had plans that day to hang out with a friend. Caught me on my way out, Said we should hang out and exchanged numbers. I didn’t do good with being caught off guard and on the spot so I said yes. He said tmr or Friday. This happened Monday. He texted me later how was it hanging out with my friend. I didn’t reply til the next day. And then asked any plans and I said just working. I was at the other building so he asked where. Seems like he keeps asking my things, haven’t replied since tho. It’s awk because he sees me on my phone all day at work. He has asked before and it was months ago, I said sure I’ll try to find time and have been busy. But now all of a sudden I said yes. I rlly dont want to. I also think he has a wife not sure.

What’s a good and nice way to say I can’t? I know it was my fault for even agreeing, but felt weird after always saying I’m busy and then all of a sudden going out to hang with a friend that day and not even hanging with him.