I cant stop having sex with my ex.

We was together for a few years and took each other's virginity. Now we are not together but still find each other having crazy sex together. Its like when we are just friends I find him way more attractive. Its not that I don't want to move on its just Im to scared  to do it with anyone else because he's the only one I ever gave my body to, this prevents me from trying to build a relationship with anyone else . Any advice ??????