Hooking up

So while I was in a relationship this guy that I knew would always pop up. My bf at the time would ask about him and I’d tell him there was nothing to worry about and there really wasn’t. I was attracted to the other but I shut it down when he asked to see me and hang out because I was in a relationship and after that the guy stopped trying. My bf and I broke up and a month later that same guy started asking if I was free and so we hooked up. My ex and I started catching up recently and he asked if I had been with anyone and I told him that it was the same guy who had been popping up. My ex is super angry and won’t talk to me at all. I understand that he’s upset but does he really have a right to completely ignore me? I mean we weren’t together when it happened and I would’ve never cheated. I just felt lonely my bf and I were together for 3 years and I was devastated. Thoughts?