Mad at him but am I being irrational? jealous and fear all in one

I feel annoyed and like I want to cry. I can't even put my finger on what it is I'm upset with.
So here's what's happening... 
My boyfriend got drunk last night and forgot he talked to me.
No biggy, I have had to many beers before. He works hard and can let lose one evening.
But today he was hungover.
I was supposed to come over and hangout  since I haven't seen him in a while.
 I knew he was hungover and figured he needed to get some sleep so I offered to come over Friday instead.
Besides, I rather use my babysitter night on an evening he wasn't over tired.
Well the day was fine. I hung with his mom and made appointments for our dogs to get groomed. Was a good day.
Then he gets distant on text. He went to his buddies house. Which I knew he ended up drinking again.
It's almost 9 and he's still there. 
I guess I feel jealous. 
I know it's irrational. 
But I thought I gave up a night to spend with him so he could recover and go to bed. 
He also spends every Sunday with his friends, drinking and hanging out.
I part of me feels worried. Is this what is to come if stay with this man? 
I guess there's underlaying concerns. 
I also get jealous that I can't just go to his place when I want. 
I have two kids. I get jealous that I can't just go hang out with him or go to my friends whenever I want. 
Why am I being so irrational? 
He's allowed to hang with his friends. 
Ugh I don't know.
I'm scared to say how I'm feeling because I don't want to make him feel wrong when he's not. It's just a reaction I'm having and can't figure out why. 
He does know I'm annoyed, said he could tell but I said it was nothing