Brother-in-law trouble

My husband's brother keeps telling their family that I hate them and now all of their family doesn't like me. My husband and I are constantly arguing because of his brother starting crap. I feel like he is trying to break us up. I told my husband that I no longer want to go over there because I know they don't like me so why spend time with people that don't like me... he told me that he spends time with my family so I should just suck it up. But I see it as they all dislike me and talk bad about me so why put myself in a situation where I'm unwated? I feel betrayed that my husband doesn't even stand up for me to his family at all. I don't know what to do, we're on the verge of separating. TIA for any advice.