Do I have a right to be annoyed?

I have been with my boyfriend 5 years.

In all these 5 years whenever we get together with friends he refuses to drink alcohol. Then when he catches up with his mates on his own he gets absolutely plastered and smashed to the point of vomiting and not remembering things.

I mean it’s fine for him to get drunk but he won’t do it in front of me. I asked him why and his response is because he thinks I’ll “tell him off” which quite annoyed me because I am always said to him have a good time and he has freedom.

This may sound silly but I feel a little bit upset and I feel like maybe I’m not good enough or something I don’t know. It’s almost like he’s hiding something. Do I have a right to be annoyed?