Is this a red flag?

So I’m totally against looking through my partner’s phone, I think it is an invasion of privacy and wouldn’t want him doing it to me. But recently, I felt super curious so I took his phone and went into his messages. There I found a conversation between him and an ex from a couple weeks ago. Apparently they had met up for coffee when I was away one weekend. He sent a text saying “thanks for meeting up with me. You look beautiful, as always 😊” my heart sank when I read this. Should I be concerned? Even if they just got coffee, he kept it from me. I confronted him about it and he said he only thinks of her as a friend, but the fact that he called her beautiful makes me think he was flirting with her. Do I have the right to be mad? What should I do?