LONG POST. In law problems!

My in laws are always talking bad about me. I got my husband out of cocaine use and alcoholism on my own. It was so difficult to make him understand that his family was part of the problem. His dad drink from Friday-Sunday the moment he wakes up till he goes to bed. His younger brother and brother in law would both give him cocaine behind my back. I slowly started telling and showing him he needed to keep a distance from his family and I felt so bad but I wasn’t going to ruin my family. He finally became completely drug free and he controls his drinking way way more now than before. We have 2 kids and I’m currently pregnant. A few days ago I found out my MIL was saying that I never let him drink with anyone and that he has anger issues because of me and a whole bunch of other stuff. A few days after my BIL was saying that I was going to backstab a friend of mine by trying to take her man like I did with my SIL which is not true at all! He said that I made his ex gf change because I wanted her to control him like I do with my husband. I’m really upset because I’ve been nothing but nice to them these last few months even after all the crap they’ve pulled. We go almost every weekend and when we don’t it’s my husbands choice. I try to stay away because they always have arguments and I obviously don’t want to have my kids around that. After this situation I told my husband he’s welcome to go whenever he wants but myself and my kids will not be going for some time, he told me he wasn’t going to go because he was upset that they’re talking about me the way they are. Am I wrong? I’m not telling him not to go but I’m not letting my kids go knowing his dad he’ll be talking about me in front of my kids or god knows what would happen to them if they go without me. I don’t trust them one bit.