What to do with him

I told this guy that I couldn’t do the casual thing anymore after sleeping with him a few times.. he said he understood but was not ready to settle down and I said I understood as well. I made it very clear and did not sleep with him the last time we hung out but lately he’s been asking me to come over really late, one time at 3 am and the second at 12 pm. Not sure what to say because I feel like I made it clear but I still would like to hang out with him but I know that’s not what he wants..

he’s also told me in the past we could be friends but also has never made any effort to hang out or talk on a platonic level