Confused

So, I have this friend. He and I really just always been sex buddies. Like if I want to have sex I'll just call him and he does the same with me. And it been like this for a while. We still have our personal lives but don't worry we're both single. Problem is I think I'm starting to like him more than just "friend". But I don't want to tell him because it's like we have our own lives and just talk when we want sex from each other or see each other outside because we're neighbors. Should I tell him or back off for a bit until this feeling goes away?