Should I feel guilty?

So there's a guy at work that I've been friends with ever since I started there (2 years ago). He knows I'm pregnant and he's cool with my husband (husband works there too), but he still recently confessed to me that he's liked me since I've been there. I made a joke out of it and said something like "too late buddy, hubby got here first" and just let it go. I don't befriend many people there and didn't want to lose him over that. Anyway, I became pregnant and hoped THAT would change his mind but he's only become more open about his feelings. It got to where he will walk past and wink at me or say flirty things to me. I got uncomfortable but didn't do or say anything. Well last week he was standing there talking to me and out of nowhere just said "I just want to have you one good time", I choked up and right as i was saying "that cant happen" my husband came around the corner and asked what we were talking about. The guy changed the subject and casually slipped away but my husband was pretty pissed off at me. I dont think he heard what the guy said but just the fact that he's noticed him talking to me more lately. The next day I told that guy not to talk to me anymore and to keep away from me and my husband as best as possible. But I feel so guilty that I didn't tell my husband what his intentions were? Do I have reason to feel this way or is it not my fault?