my bf gets mad that I don't trust him

So he's done things before to ruin the trust I had in him, such as texting other girls and being flirty. I didn't find those things out through him. It's always stayed at the back of my mind because I didn't have any idea that was happening and it could still be happening and I'm in the dark about it. So I check his phone sometimes just to reassure myself since he doesn't do it. He gets mad because he says I do it too often and that means I don't trust him. But I have a reason not to trust him but I don't want him to think I don't cause then he'll say "so why are you with me" and everything else is fine, but he doesn't do things that reassure me he's faithful