On/off dating for a year?

We were never official. We would hang out and go on dates. The first month of us dating he told me he likes me, but his feelings aren't as strong as mine. I told him it's ok and thanks for being honest. I stopped talking to him after that.
Couple months passed and he randomly texts me at 2 am. He tells me how he missed me and liked what we had. He wants to try it again. We see each other in person and he tells me how he is sorry if he hurt me. He tells me how we couldn't work out because of his dad(?). We ended up messing around a little bit then it went back to us not talking again. 
Of course he talks to me again a few months later. We went out and then finally ended up having sex. Then I finally stopped the contact with him. For the next 3 months it becomes him contacting me and us having sex each time then us not talking anymore. Now he wants to try for something real and not just sex. I just don't trust him because I feel like I was just a side piece. I know I should of stopped having sex with him, but it was too good. It seemed like I was only there out of his own convenience. What should I do?