Husbands brother is driving me CRAZY!!

My husband has this douche of a brother. Every time we go over his mom’s house his little brother is SO DISRESPECTFUL to him. He calls him a bitch in front of me, says he’s broke, lazy, etc. He complains about EVERYTHING and tries to make my husband look stupid in front of people. Since the first day I met him I knew he was a jerk so I’ve never even spoken to him. I feel like he’s disrespecting me by disrespecting my husband in front of me. I told my husband it bothers me and his parents told him something but he still does it. I told my husband to stick up for himself but he likes to avoid confrontation. I’m so close to calling him out but I feel like he’ll get upset and turn into something bigger. Do I have a right to say something or should I stay out of their personal brother issues?