I have a question I really need answered..

My boyfriend knows everyone I've ever been with, that's because they've all been from our home town. But I don't know everyone he's been with.. I know that when our relationship first started, him knowing everyone I've been with really bothered him because he use to picture them with me or wonder if they were better... My question is if I should know his too so it's even? He's said if I want to know than he will tell me. So we are both emotionally equivalent of knowing each other's pasts...