Need genuine advice

Hi ladies!

So I’ve been dating a wonderful guy for almost two years. We have a super trusting relationship and he has done nothing to ever break my trust. We’re not the types to ever go through phones or ask around but I know he’s a good guy and hes loyal to me. He has a couple girl friends who are awesome girls and I never feel any pang of jealousy when they hang out. I have my own friends and guy friends and as far as I know he’s always felt fine about it too. However there is one girl I get jealous about.... his ex girlfriend. She’s a nice girl and I’ve met her and I know she is kind. It’s his only other girl he’s ever dated and I know she was his first everything. They’ve known eachother for ever. They have the same friend group so I knew they would hang out. He asked when we first started dating if it was ok and I said of course! And he tells me every time, he even invites me when he can. Well now I live across the country and we are long distance for school. I STILL want to feel great about them hanging out and I feel super guilty for even being uncomfortable with it since I literally told him he could any time. I dontwant to be the girlfriend that needs to give permission. My problem is that I know that he’s loyal and I know she’s a nice girl but they are drinking together right now with friends and I just get nervous that he’ll get drunk or something. I feel like crap for thinking these things. I am guessing it’s stronger because we’re far away and I miss him. They’ve just hung out like 4 times this week. Ugh!! Any advice on how to get the heck over it!!!