fwb guilt

So I've known this guy since my freshman year and we're close friends, we smoke and hangout, we have a lot in common, but he's always had a girl friend as long as I've known him. A few weeks ago him and his girlfriend (one of my friends) broke up. The guy and I were hanging out and ended up kissing:/ it just kinda happened and it was like an instant spark and since then we've become fwb but I feel so guilty cause it's one of my friend's ex boyfriends. But I've been friends and closer with the guy longer and his ex and I aren't that close we're more friends with the same people if you get what I mean. Should I feel guilty? Should I end it? He makes me happy but I don't want anyone getting hurt