Husband crossing the line?

My husband said he is depressed over me not allowing him to bring our baby to his family gatherings. Me and his mother are not on good terms. She is a white woman and I am Asian. She has made racist comments in front of me while my husband tries to laugh it off… He claims that he defends me in private with her but can’t do it in front me… our daughter is half Asian and since she was three months old I had cut his family off from my life and our daughter’s. He claims that his family has a right to see her because she is their granddaughter. His sister and dad were good to me and he said visits from them isn’t enough and wants to bring her to his parents house where his mom would be involved. I don’t feel comfortable with this and don’t know what to do.