Are we wrong for not trusting my husband's dad with our kids

My husband doesn't really talk to his dad a lot. He will call sometimes but for the most part they don't have a big relationship. His dad has met our kids but he hasn't been alone with them. My husband's dad asked to have the kids for the weekend and my husband said no. He got upset because my husband allows his ex step mom to have the kids for the weekend. My husband adores his step mom. I love her too.

We don't trust my husband's dad because he's an idiot with a lot of money and those two things don't mix. His dad is pretty rich and would have parties when my husband was a minor and when my husband was 16 he gave him coke. My husband describes it as him pushing it towards him and telling him to snort it. So he got addicted to coke at a young age and when he was 17 and his dad was having a party. He was extremely high on the couch and got sexually assaulted by a 20 year old woman from his dad's party. His dad still refuses to acknowledge that he did anything wrong or that my husband suffered any trauma from his S.A. He actually thinks he did my husband a favor because that's how he lost his virginity which is gross.

He does apologize and still continues to live a wreckless life with inviting random people in his home. Him and my husband are not close so why does he think when he has shown he's an irresponsible father that we could trust him to be a responsible grandfather.