I don't want my boyfriend anymore

We've been together for 4 years. About 2 years ago I've completely lost any interest in having sex with him.. Zero sex drive. I thought maybe I'm lesbian or asexual or something like that.. Our relationship became miserable since that time. But still I loved him and hoped it will get better.. Until I had affair with another guy. That was whenI Realized  that I am totally fine, I just don't want my bf anymore!! Now I even don't want to be touch by him :( ... Is it time to break up? What do I tell him? I don't want to hurt him.. Advice me plz..