Visit - yes or no?

My children's father hasn't visited them in 2 months since moving states, his mother who didn't see the kids for about 2 years up until Dec 2020 for the first time wants them to go to her house during his visits. Would you allow it? I'm conflicted because she hasn't been the best person in the past and wanted nothing to do with my children until I divorced their father. They went to her house for the day once since he moved and were crushed he wasn't there when they went to her house and came back feeling devastated their dad wasn't there to visit them. She wants them for the day again, but I don't think it would be good for them since they haven't seen their dad in 2 months and he hasn't even reached out to ask about them or call to talk to them. Would you let your kids visit again without their dad there?