Feeling left out..

My SO and his brother are really close and in the beginning of our relationship we hung out a lot. My SO, me, his brother and his girlfriend. Now we don't hang as much but somehow it always ends up being my so, his brother and his girlfriend hanging out. I feel really left out and just like crap. I told my SO and of course ya know he says I'm not and not to feel left out. But this isn't the first time it's happened. And it's not the first time I've brought it up to my SO. He says when they hang it's supposed to be him and his brother and the girlfriend always comes home and they end up all hanging out. Right now they're all together and I just got off work and I just feel so left out.