Could I be overthinking

My bf/bd cheated years ago. He hasn’t cheated since that I know of. But he knows I still struggle with trust a lot. It seems he’s getting annoyed by it now. Like when he’s on his phone sometimes I’ll just stare at him and just sit there and wonder who tf he’s texting. If he catches me staring at him he gets mad and claims it’s “every time he’s on his phone”. Bc he got mad now I’m thinking he is cheating. But another part of me thinks I’m wrong and that he has the right to be annoyed by it. What do y’all think? Be brutally honest.