ex left

My babies dad left me he said he didnt care i couldnt do nothing right not even have good sex n stuff n he told me to leave so i did. But the thing is he always calls me after like a week or two saying he misses me or he comes to my house and tries to apologize n i go back. what should i do to not go back bc he always ends up making me go back n manipulating me 😢