Grandma's Boy😑

I really need some advice!
Ok so my husband and I have been together since high school to make a long story short he's white in black. He was raised by his grandmother who I later found out after we got married that she doesn't like black ppl! So all the time spent around her during our high school/ college years she was being fake! Anyway when I got pregnant with our son she seems "happy" for him at least, around the time I was 6 months she totally flipped the switch and I mean went total rascist b**** on me! We were visiting at the time and my husband went to hang with some friends and she took that opportunity to curse me out call me the "N" word and called me a s""" and every racist thing you can think of! I called my husband that and we left that night. He didn't confront her until a few days later about it. Few days after they're all buddy buddy again calling her all smiles and laughter! And I felt like he didn't defend me of his unborn child at all. But he ensured me she said she was wrong and blah blah that was 2011 and until this day 2016 she Has never once apologized. I've since cut her from my life my son got to know her because I chose to higher road! I hate it when she visits but I put my feelings aside for my son because I don't want to grew my child with hate and disdain. Ok to the point. We're in a good place in our live right now better than before. We are considering vacations trips to places with our son. I want to make these memories our own with our child. And every time we make plans my husband wants her to come along! Like wtf!?! Am I being selfish here? I've expressed how I felt about her I've said it over and over holidays like thanksgiving and Easter is fine she can come stay a night and leave...but why does he want her on vacations why does he wants to drag her around with us? And then he says crap like we can't go anywhere or do anything then! It's like he's forcing me and I don't think that's fair at all! Any advice would be so appreciated I feel like i need to speak with someone else about this!Â