Small town & nosey ex

I live in a small town where everyone knows just about everyone . Me and my ex of 6 years broke up last year and there was one guy I spoke to a few times and my ex heard & abused us both to the point he scared the guy away. Just a few weekends ago I met a guy out on the town, we were hanging out at the pub and we exchanged numbers, we've been talking ever since but just tonight my ex has messaged me to let me know he now knows and as manipulative & emotionally abusive he is he will always have some crap to say. I just don't know how I am going to live in a town where everybody knows & nothing can be kept on the downlow.. I also know I need to not give a damn what my ex says or thinks, but he is very toxic and will mess up whatever he can with whatever chance he gets Including any new guy I may communicate with😔