Desperately attracted to a co-worker!!

Okay so I’m having a problem. I have a deep attraction for one of my coworkers. His smile is gorgeous and he always makes a point to come over to my department to talk to me for Atleast 10 minutes a day. Problem? I’m married. I’ve been married for two years and me and my husband are kind of going through it right now. I’ve never cheated on anyone, and don’t intend to, but I think about it often. My coworker has made advances and I’ve turned him down every time. One day he came and said very to me in a very sad way, I asked him what was wrong and he said “ah nothing, just lust” 😭😭 I think I’m gonna have to get a new job. Any advice for me? Should I just stop talking to him, or get a new job entirely? Also should I talk to my husband about it? Any advice is welcomed.