Advice

So yesterday my boyfriend's brother asked him to smoke with him at his ex's house and asked if I would go to and of course I said no and didn't want my boyfriend going either since ts his ex, we got into a huge fight over it. I find it disrespectful that he wanted to go and did go due to that being his ex and he's cheated in me before. I just don't know what to do anymore. What would you/have you done?