Jealous of his best bud...

Ashley • I'm 30, have an amazing husband, and we have a beautiful baby girl together.
So my boyfriend's best guy friend lives kinda out in the middle of nowhere and only comes into town every so often. Whenever he does come to town, the guys go golfing or for drinks or something. They've been best friends for like 10 years, and I love that they get to go hang out, but he never tells me about it until he's pretty much walking out the door and I never get to go hang out with the guy. I've only met him one time (in about a year), and they hang out about every 3 weeks or so. It makes me feel like a third wheel whenever he comes to town, and I get resentful. I hate feeling like this, and whenever I bring it up I end up feeling bad because my SO and I pretty much do everything together and this is his time to get away and just be a dude for a while. What should I do? How do I get past feeling jealous of his best friend??