“love you” too late

My ex had a really bad relationship with his ex. She was controlling to the point where she would get jealous over his own family, didn’t let him talk to his family for almost a year & he loved her/wanted her happy, he prioritized her. I met him about 3 months after he broke up with her & we started dating probably 3-4 months after. His ex was batshit crazy, would contact me and my family saying he was a cheater and that he was going to cheat on me. She also told my mom they saw us coming out of a hotel (which isn’t true but was trying to make me look promiscuous). She also told them I was pregnant (my mom already knew the story so ofc she didn’t fall for it). Then, she was posting on Facebook saying he was a pedophile and dating a 15 year old (I’m 20 next month but I look young lol). Well, after years of being with her, I understand that he might’ve been on the edge throughout our relationship, but I was getting more and more disappointed after many chances. I’m back in college, 2 hours away from home, and we talk occasionally. I told him I didn’t love him anymore but I did care about him still. He then sends me this message

How do I let him down easily? I don’t want to break his heart because I know he genuinely cares about me. He’s done SO much for me throughout our relationship, but the fights took a toll on me. The things he’d tell me took a toll on me. Slowly I was losing feelings for him & the last fight we had was the final straw. Any advice?

Edit: we broke up because of both. I understand that his ex was kind of psycho (she’d still post old pics of them together) but I tried to be understanding because I dealt with an ex like that. I gave him chance after chance to work out his issues but he’d fall back into his insecurity and negative mindset