Monster-in-law?

I usually don't post on here at all, but I've scoured the internet high and low and havent found any solutions or advice on my problem. First off, I'd just like to start by saying I dont actively want a divorce, I'm hoping for some advice or some ways to strengthen my marriage and get past these issues if at all possible. My husband and I have known each other since we were 14 years old and have always been pretty good friends, when we decided to start dating and eventually got married everything felt like complete bliss. We have had several arguments, disagreements in the past and normally my husband and I can talk about ANYTHING that is bothering us without any hesitancy at all, we have very open lines of communication. EXCEPT when it comes to his family and the way they treat me. The first couple of years, I was very close to his family, having gotten along with all of his siblings and mom very well. His mother's boyfriend however is the only exception, he's very creepy and is always making sexual comments, even towards/about younger teenage girls (i have my unconfirmed suspicions as do the rest of his siblings). The rest of my husbands large family just seems to brush it off or sweep it under the rug to keep things okay with their mom. Several years ago, at his family's Christmas party, his mother's boyfriend, who is an alcoholic, grabbed my arm hard enough to bruise me, and my husband had a fit(needless to say I was furious as well). His mother, however, felt as though the whole thing was no big deal and the 2 of us were just overreacting, "he was drunk, didnt know he was doing it. etc...." Trying to avoid him and protect myself, I avoid going over there as much as possible, including limiting my time even on holidays. Obviously, this has put a rift between me and my mother in law, and although my husband doesnt directly say it, I know it is straining our marriage. Ever since I've decided to distance myself from them, his family has continually made comments about me, especially his mother, both directly to me and behind my back, which isnt helping me to want to work on things with them either. lately, I know my husband is getting fed up with me not wanting to be around his family, and I know personally im not sure how long I can keep pretending that spending holidays apart isnt bugging me. My main reason for needing the advice is that for the past couple of years we have been TTC, but after this past Easter, Im starting to see that things will probably never change and I would feel selfish not allowing my kids to see their grandparents or spend holidays with that side of the family for fear that my husband wont watch them closely enough around his moms boyfriend, and I won't be there myself. I've tried to discuss this with my husband, but it seems like he just shuts down and avoids the topic altogether. it's getting to the point where im not sure my marriage is going to make it, I really want to have a family, but cant deal with my in-laws enough to be stuck with them forever, which really hurts because i love my husband so much. I guess im just needing advice on how to work through it? Or what i should do next?