Friends With What?

Had sex for the first time with a friend I have cared about for a while. he is on a break from his relationship. We both agreed we shouldn't let it happen again and but it has happened twice since the first time. I have all these feelings for him, don't think he feels the same  but I don't want to scare him away, what should I do?