He likes me but doesn't want a girlfriend?

What the title says... I'm trying hard not to but I really like this guy I use to work with. He's a bit younger than me but we get along really well. We don't hang out often but when we do it's the best time I have in a while. He tells me he really likes me and he acts like it when we're around each other but then he says he doesn't want a girlfriend and we will go weeks without talking. I tell him I think he's full of it when he says he likes me (because we have slept together and he is a dude) but I can tell it bothers him a little. I'm not really sure what to do here, wait it out or let it go? I don't want to nag him or make uncomfortable. Thoughts?