My family said my husband isn't allowed at the 4th of July

My parents told me today that it hurts them to say this but my husband isn't invited to the 4th of July because my grandparents don't want to be around him. My grandparents started falling out with my husband a while ago. My family is Catholic/christian and my husband identified as Christian when we got married. Over the last several months my husband has been converting to Judaism and has been working with a rabbi to do so. I may be christian but the religion of other people was never important to me. I officiated my athiest gay best friends wedding so I obviously wasnt bothered. But my grandparents were and they were upset that he was converting so publicly. They think my husband will damn all our children to hell if they decide to convert to Judaism too. So things have already been tense. Why they're so upset at him now is because they found out my husband participated in this program 2 years ago and answered questions for an article. The point of the program is to bring light of child sexual abuse in the Catholic church and my husband did an interview with a journalist to give his story on his childhood sexual abuse within the Catholic church. I'm the only one who knew he did it but it was found by another family member and send to my parents and grandparents. My grandparents were upset and said my husband was continuing this stereotype that priest molest underage boys. It was something he wanted to do and if it helped him heal I supported him. My grandparents are upset with my husband and want an apology for being so public about leaving Christianity for Judaism and for him participating in that interview. I told my husband he doesn't have to apologize for anything. Now I'm being told that me and the kids can come but my husband is not allowed. My husband was hurt but told me and the kids to have fun. But I don't want to go without him. I know the kids should be able to see their family but I just don't think this is fair.