Friends with benefits

So I'm having a relationship with my friend. He has a girlfriend but we have always been attracted to each other even before he got a girlfriend. So I'm the other women. The thing is I want to stop but it's hard. Aside from us having sex which at times we use a condom or not he pulls out. He is my Close friend. So I talk to him about things that I normally talk about a friend. We hang around each other but always in secret. I want to stop but I cant. Any suggestions on how I can stop without having to lose my friend?