Edit: Small fight don't know what to do

So husband and I are fighting because I want to take my two kids and stay at my parents house tomorrow night. He doesn't want me to. I want to go because his two kids come over tomorrow for just one night. I love them dearly and my kids love them but their mother makes stuff up Everytime they go home. My husband says if we leave it makes it seem like we are hiding something. I say that she can't say things happened if we aren't here. It's been stuff like I beat her kids then my daughter beat her kids then her kids were sexually touched and it must have been my toddler that did it and forced them. None of this was proved or even brought to us by any legal person. I just can't do it. I'm almost 6 months pregnant. I think it would be nice to not have to worry about it, he disagrees and it's making us fight. I don't know what to do. Literally as soon as they get home she messages about a small scrape or a bruised knee even though we tell her about it before they go home. I just can't take it. Especially being pregnant she's driving me insane. She's telling me I'm making her kids sick because they can't have dairy (not true I have doctor's proof) but she said they can't have it but they can have cooked cheese??? It's to much stress and he just doesn't get it. He's worried about making ourselves look bad plus he wants us all to be together as a family. Help?

Edit: it has nothing to do with limited food. As I said the boys ask for things that she gives them..chocolate milk and such so husband asked there doctor about it. They have no allergies. No reason for them not to have it. She wants people to think we are making them unhealthy. I said a doctor told us it wasn't true. I'm not referring to them as his and mine to make it worse. They are mine but to make it more understandable biologically they aren't mine so I can't do much and yes these accusations are serious. The point of this