Mad at him but am I being irrational? jealous and fear all in one

I feel annoyed and like I want to cry. I can't even put my finger on what it is I'm upset with.
So here's what's happening... 
My boyfriend got drunk last night and forgot he talked to me.
No biggy, I have had to many beers before. He works hard and can let lose one evening.
But today he was hungover.
I was supposed to come over and hangout  since I haven't seen him in a while.
 I knew he was hungover and figured he needed to get some sleep so I offered to come over Friday instead.
Besides, I rather use my babysitter night on an evening he wasn't over tired.
Well the day was fine. I hung with his mom and made appointments for our dogs to get groomed. Was a good day.
Then he gets distant on text. He went to his buddies house. Which I knew he ended up drinking again.
It's almost 9 and he's still there. 
I guess I feel jealous. 
I know it's irrational. 
But I thought I gave up a night to spend with him so he could recover and go to bed. 
He also spends every Sunday with his friends, drinking and hanging out.
I part of me feels worried. Is this what is to come if stay with this man? 
I guess there's underlaying concerns. 
I also get jealous that I can't just go to his place when I want. 
I have two kids. I get jealous that I can't just go hang out with him or go to my friends whenever I want. 
Why am I being so irrational? 
He's allowed to hang with his friends. 
Ugh I don't know.
I'm scared to say how I'm feeling because I don't want to make him feel wrong when he's not. It's just a reaction I'm having and can't figure out why. 
He does know I'm annoyed, said he could tell but I said it was nothing

Glow Resources

Let’s Glow

Glow is here for you on your path to pregnancy

Glow helps you navigate your fertility journey with smart tools, personalized insights, and guidance from medical experts who understand what matters most.

25+ million

Users

4.8 stars

200k+ app ratings

20+

Medical advisors