Boyfriend keeps looking through my phone

When i come over sometimes he grabs my phone and goes to “just scroll through instagram” and then he goes to my messages and looks through them in front of me. I confront him every time he does this yet he still does it. i tell him to stop and i make it a point to tell him i’ve never looked through his phone, which he responded “but you can whenever you want” i told him i’m not ever going to because i trust you. He then says he trusts me too, it’s other people he doesn’t. he’s stated a bunch of people always hit on me, which is true, but i always show i’m not interested whatsoever. Do i threaten to leave if he doesn’t stop? Because this means he probably doesn’t trust me right?