Kids and grandparents

Does anyone else not let there children have a sleepover at there grandparents? I mean my daughter stays over at my moms but not his parents. I mean they never really asked for them to sleep over but seem mad when they see us and the kids are with my parents. I don't really feel comfortable with them being at there without me.. I just want to know if this is crazy. Am I wrong for letting them stay one place but not the other?