Checking his phone ?

Okay so I’ve been with my boyfriend for about two years now. And I’ve always grabbed his phone to change music or send pictures we take on his phone to me... or do little things like that. And he does that with mine and we’ve never had a problem with that.

He recently saw me snooping through his phone while I was changing the music and I’ve done it in the past and never gotten this mad. I told him that and he said he’s never liked it and that he feels like I shouldn’t need to because Ive never caught him doing anything wrong. He has got upset because he says he feels like I’m invading his space and it’s just his phone so he doesn’t feel comfortable when I start looking through his things. I understand that but my opinion is why does he get bothered unless he’s doing something wrong? I don’t get bothered when he wants to click on my messages or go my social media. I do get a little uncomfortable but not to the point where I wanna say stop looking through my phone.