TTC and having to deal with poisonous people

Emily
In brief, straight up, my husbands brother is a douchbag. My husband, my husbands brother and mother in law don't even like him. Half the time I don't think his wife even likes him!
He's rude, offensive, insensitive, disrespectful manipulative, condescending and a total control neat freak. We are constantly buying heads because as my husband says "we're on total opposite ends of a personal spectrum". He has verbally attacked me once in private, totally talks crap about me when I'm not around and manages to insult me everytime I'm around him.
I'm TTC and I'm currently off anti anxiety and depression meds to clean out my system. I've told my husband and he completely understands that my BIL is poison and I should not be around him because he's only upsetting to me and can only harm my TTC.
Unfortunately we live about 15 mins from him and are new to the area so we dot have much of a social life outside of them YET and because my BIL is so vile, he doesn't have his own friends to hang out with and is always trying to get my husband to hang out.
What do you think I should do? It's already driven a wedge between my husband and me even though my husband completely understands. 
I