I dunno what to do

I have an ex that everyone in my family and friends despises. We broke up because we were both super stressed. He gets clingy when he's stressed. I need space when I'm stressed. This turned into an argument and then a breakup. My friends and family all hate him because he's "too clingy." He's my best friend though. We laugh and argue and he's wonderful to me. He still loves me, and I still love him. I don't know what to do though...everyone hates him and I don't want to listen to all the yelling if I get back with him. His friends and family also hate me because they think I'm too distant and overdramatic. I want to be with him, but I don't know how to do that without everyone criticising us...