Am I making the right decision

My boyfriend and I met 5 years ago at church. At the time I was in a very abusive relationship and couldn't get out just yet. I finally left him and in that time my current boyfriend moved back to Georgia because his mom died. We spent years searching for each other so last year we reconnected. We've been together since. We lived in florida for a while and then we moved back to my home state. Since being back home hes been so mean to me calling me a whore and acting distant so he went to visit family in Georgia but I don't want him to come back. I'm doing good without him and im happy and he's nicer to me being in Georgia than he is when he's here with me. Would it be wrong of me to ask him to stay and just break up with him or should I stay? We don't have kids together that are alive. I had a miscarriage recently