Am I horrible for doing this?

Am I horrible for letting our baby go to my mother's all of the time staying over night ect but not letting my partners mom baby sit or have the baby over night? I get along with his family but I don't know if I could trust his side of the family to have our baby or have our baby over night. I feel awful but I know if my partners mom had the baby over night I wouldn't be able to sleep or anything. I think I'd be more comfortable later on when the baby is older but right now the only person I can trust alone with the baby is my partner and my mom/dad/brothers. Is this normal? Tia x