I don’t trust my in laws

My husband gets upset that I completely trust my family with my baby, but i don’t trust his family. My baby girl is my first and only child. I don’t trust my mil because I saw her shaking my baby when my baby was so little I had a c section so I needed help taking care of her. When I saw my mil shaking my baby I decided to ask my mom for help instead of my mil. Plus my husband has two brothers adults and they live with his parents you could never trust people nowadays I hear so many scary stories so I chose to keep my baby with my or leave her with my family. We argue over that almost everyday he says his mom didn’t shake her angrily like why would you even shake a baby angrily or happily. I let them visit us she could play with them but I have to be in the same room. Am I wrong here or what?