Wanting something more??

My co-worker and I are good friends but everyone is telling me he is interested in me. I feel awkward towards him and I don't date co-workers since it will only cause problems. He told my boyfriend that "he doesn't date other guys' girls" yet he wants to hang out at his place alone. Any advice? 
(Have a boyfriend and am poly. Although not really looking for another partner)