red flag ? should I waste my time

So I've been dating this guy from work. I wasn't going to get involved at the start because he was fuck buddies with my coworker. However word got out that he tried to hit on me and the girl cut him off for good. The thing is I went on vacation and he thought I got a new job. He said his intentions were to take me out from the jump. However things got in the way. However it seems like the way everything has happened between relied on timing. One thing let to another he asked me on a date. We work together and have been on like 4 dates. He is a nice guy, however is is older by a few years. Anyways, he seems to bring up his ex a lot and even compared us. Which I found really weird and a red flag that he isn't over his ex. I asked him how long they dated and he said 1 year and nine months. On and off. I'm really starting to like the guy, he have a great time we go out however something doesn't feel right about it. Anyone else would be concerned ?