How to reject someone after agreeing to hang out and go on a possible date?

I work as a concierge taking care of people living at a luxury building. I’ve been working there for about two weeks and the first week I met a resident who would not stop talking to me and then asked to go out and get food right away. I was just being nice and said yes I would be down to. I didn’t think anything of it. There’s two buildings I’m at but they’re separate but close. He seems to come to both buildings and is always lingering around and chatting with me a few times in the day. Even if I’m at the building he’s not living in. Everytime I see him, he asks to go get food but I’ve been making excuses. I just get weird feelings around him and do NOT want to hang out with him. My fault for saying yes in the first place but how do I tell him I would not like to go in a professional way but keeps things good between us? He even offered to buy me a muffin while I was working, I said no but he still came back with one for me. I asked a friend and she told me to just say I have a boyfriend but I do not have one. I overthink and think about how he might ask me for a pic of us together. So idk if that might work. What do you guys suggest I tell him? I do NOT want to be reported if I say anything wrong and possibly lose this job. I see him every day I’m at work so I’m not sure how to go about this.