I need advice :(

My 3 year old son does not like my parents. It hasn't always been like this! Pre COVID he would spend the night and they would watch him for a few hours several times a month and he seemed to enjoy being there and seeing them. The covid happened and we didn't see them for a long time. At some point when we did start seeing them again, he would say things about how he didn't want to go over there, but we would still take him over there and visit as a family, and most of the time even if he said he didn't want to go, he ended up having a good time. But it's gotten worse. Cries when we get to their house, yells that he doesn't want to see them. It breaks my heart and it breaks theirs. They are very passive people, they don't yell. I think they're great parents and grandparents and for the life of us, we can't figure out why he is acting like this. He adores his other set of grandparents, which makes it even harder for me that he's decided he hates mine. I struggle because obviously seeing them is causing him stress, but I can't just not see my parents because of my son. We've tried just doing video chats and he still hides behind his hands :( I don't know what to do. I'm terrified he will grow up and not like or love his grandparents.