Going out to the bar without me

My boyfriend and I have been together for almost 4 years, we are both 22. Since I've turned 21, he's never gone out to the bar with me. However, he will go out with his Fraternity brothers. Whenever there are small parties at his Fraternity house, he never comes to hangout with me; he will hangout with brothers or in his room. He says he doesn't like me when I drink, but has only been around me a few times.

Is it weird that he enjoys drinking and socializing when I'm not with him?