I need answers

Question in a situation where it your Man or your Family which one will you choose? Now let me explain what I'm talking about I got man loves me and cares for me and that wants to be with me. But he don't feel comfortable with out situation I have my own house and I have my brother and his family who had been staying her for 3 yrs and it's like they don't want to leave. I have sake when they moving out I gets attitude because I asked a ? About my house I'm just so sick and tired I don't want to call the cops because I don't want to involve them in this situation I just want my brother to understand that I want my house to myself me and my man??? What should I do I need answer I have made my choice I'm choosing my MAN OVER MY FAMILY!!!!!!!!!!' Reason being is because I don't have a relationship with my brother and he's living his life the way he want and I just want to do the same. I'm in love with this man and I don't want that to every change we have so much fun together we laugh we joke we have a relationship that's not perfect but he's just right for Me!!!!!!!!!!!!!!