Is he cheating or just distrusting?

Whenever my boyfriend and I are together, we leave our phones set aside and don't use them very much. Sometimes when we are watching a movie, I catch him going through my phone! Which is totally fine, I don't mind if he does that IF he asks. But I have caught him doing this four times in the past month even after talking to him about it. I get annoyed about this because he claims to trust me! And I never ever touch his phone without asking him first- out of respect for him and his privacy... I do trust him! I talked to my mom about this and she said usually when people do this it is because they are doing something wrong and they reflect their wrong onto the other. What do you guys think? I could really use outside opinions. 
​Additional details: I have mostly guy friends and I get a lot of attention from guys which seems to make him kind of mad!