guilty from breaking his heart

So my boyfriend and i broke up few months ago, we didn't worked out bc I couldn't communicate and build a foundation of trust with him, but we stayed friends bc maybe as friends I could fix my issues but he's still there, and we were honest about how we still have feelings for each other. He said he will wait, till im ready for a relationship. Lately, as friends we had an argument, that's when I decided irrationally that I don't wanna stay friends with him anymore. He's so intellectual, he always have an opinion about everything and i just couldn't keep up with him. I do make a lot of mistakes for him. But i decided im done, he gives me way too many chances, so I thought he's too good for me. He thinks otherwise. But i told him i give up, he let me go but he told me how much this breaks him. Is it wrong to give up on someone you love?