Awkward situation

I work in a bar where mainly older married men drink, one of them has been gettin really close with me got my number from a friend and has been texting me as while. 
It started off quite innocent but now he's telling me things aren't good at home with his wife and he's looking to have a bit of fun. 
I mean I feel guilty enough that he's texting me but once he asked for "fun" I told him no straight away but he still keeps asking... He's always in the bar when I'm working and it just feels awkward! 
Any advice on what to do in this situation?
Update: my friend only gave him my number cause he was looking my brother for a couple of days work she didn't think she was doing anything wrong