boyfriends friends gf

my boyfriend always goes with his best friend/roommate to his girlfriend's house. they all chill and smoke and do crap together but it makes me uncomfortable bc i've never been formally invited and she has 3 other roommates who are also girls that i know for a fact have had a thing for him in the past.

Is it wrong of me to ask him to stay home and not go with his best friend to have a "good time"?

or am i right to break up with him if he continues to go..

(i asked him to not go and he said it's not fair to ask that of him) am i out of place?

PS. i trust him in the sense of i know he's not going to physically cheat on me, but i just don't like the idea of him being at some girls house with 3 other girls till like 1 or 2 am. be it with his guy friend or not.

EDIT: bc i wanna stay anonymous i'm going to reply here.

we have spoken about it three times in the past. i have cut off many of my male friends that were interested in me bc it made him uncomfortable, it is something i had no problem with, he says he doesn't have a problem with me going but every time theyve been at his place, i'm pushed aside and ignored for his interactions with his roommates.

so i have met these girls and i've seen videos of him dancing with them to music and having a great time and he shows no interest of wanting me to join this "great time"

he's upset bc i basically have no girl friends other than my roommates (one is never home and one never wants to leave the house) so i'm asking him to stay home and spend time with me instead of go to these girls' house. he says it's not fair to ask of him to leave his friends but he just met most of them anyways, they're not his true friends and if he wants to spend time with his best friend, they live together anyway, so i don't see the problem with him passing up a night with a bunch of promiscuous women who i have seen flirt with him in front of me.