Walking away !

Been sleeping with this guy off and on for a few months we quit talking for about 3 weeks because he wanted to see someone else ....but then he came back I told him that I wanted to date but he said he wasn't "boyfriend material" we only see each other about every two weeks and I really like this dude and the sex is fantastic but I don't think it's a good situation and I've tried to walk away twice and he has sucked me back in each time I have never had a problem walking away before ...why now ?!?! Any tips on how to be done for good ?