Don't know where to go from here

I love my boyfriend. He's smart, he's funny, he's great in bed, and he knows how to communicate with me. Well, I've always had a sense that he's not good at getting over his exes. They all seem to take a piece of him and he doesn't have much of himself left for me. Last night, he told me he knew I loved him (because he's egotistical, yay) and that he basically couldn't love me back. He then went on to talk about his first love (which he had told me about 4 or 5 months into the relationship). I'm sure many of you would be like WHAT THE HELL, but I'm fine with it. I know, it's weird that I would be fine with loving someone who states he isn't capable of loving me back. And after that conversation, I also feel bothered. You're probably like DUH. But I'm more bothered by the fact that his exes can have such a powerful impact, especially the most fucked up one, a married woman who used him for sex and then after her divorce, cheated on him (his first love 😕). I don't know what to do. I feel like if I'm not the one for him, I'm wasting his time. I love this guy, but I also love him enough to let him go. I want him to be happy. I'm conflicted. I asked him what he wanted me to do with that information and he said he didn't know. Well, I DON'T KNOW EITHER.