Am I wrong?

I have a boyfriend. Every day at work this guy comes in to talk to me. It started off very little but now he's hanging out and talking to me for like two hours. I'm stuck behind a desk and can't leave. I've mentioned the fact that I'm getting in trouble for talking too much and I tell him or mention my boyfriend every day. But for some reason he won't leave me alone. In the beginning I thought it was ok because we were just friends, he doesn't have my number, and we don't hangout outside of work. But this morning he brought me a tea and now it's crossed the line because I know he likes me. Am I in the wrong for trying to be friends? I never wanted to hangout after work and I have been firm on not giving my number. I wouldn't like if my boyfriend was doing this so I want to stop talking to him. The problem is, I feel like itsgone on too long so it'll be awkward. I in no way am attracted to him and do not like him. The conversations are ones I would have with any of them. Although, when he brought a friend they did talk about sexual stuff like not towards me just a general conversation (like one I have with my best friend). I am going to tell my boyfriend that he has been talking to me and I am going to ask advice on how to tell him to leave me alone because I want him aware of the situation. But am I wrong?? It's been like two weeks maybe that he's been coming in for longer periods of time. I feel so guilty. I can't wait to tell my boyfriend. He won't be mad but I just want to know if what I did was wrong by having long conversations. (I was in an abusive relationship and couldn't do anything so I'm not sure what is normal and what is not)