Open phone policy

In my relationship we have an open phone policy. We don't keep anything from eachother and it just works. Neither of us ever feels the need to go through the other ones phone anyway and if we wanna read the specifics of a conversation we were told about we just go look. But usually we are sitting next to eachother. I went to the rest room and took his phone (because he wanted me to look at something else...) I went on to see a conversation he had already told me about. I saw something I didn't know. He felt like I shouldn't of looked when he wasn't next to me. We didn't argue over it, but he was a little upset and I apologized because I felt I did something wrong. Did I? As soon as I saw the information I didn't know I stopped reading immediately so he could tell me about it himself.