Having our baby around my FIL

So my husband's dad is not his actual dad. His real dad walked out on him when he was 2. His mom's husband is who he calls dad and he adopted my husband when my husband was 7. My husband does call him dad but has always been a little distant with him. I'm pregnant and my MIL and FIL are excited to see the baby. My FIL is already asking when they get overnights and and MIL is agreeing.

My husband spoke to me and said he's struggling internally about if he wants his parents in our babies lives. I was pretty shocked to hear that. He seemed to have a good relationship with them. But he told me he never told me something. When he was a teenager for his 15th birthday his dad(technically step dad) gave him some alcohol. They got a little drunk and then his step dad picked him up and through him on the couch and held him down. NOTHING HAPPENED! He stressed nothing happened. But his dad while holding him down said how much he wanted to.... And how much he could right now. But that after a while of holding him down like that he got off and went to bed drunk.

My husband said his dad did apologize and they went to therapy through the church but that family therapy was less about what his dad did and more about helping my husband let it go and forgive his dad. He told me he thinks he forgives him but one thing his dad said every time he brings it up is "Tyler. If I wanted to I would have. I literally could have in that moment but I didn't." He never did anything like that again but my husband just doesn't feel like he ever got a true apology.

I'm really nervous about having this man around our baby now. I get he didn't do anything to my husband and he stepped up as his dad when his real dad left. But I was actually raped as a child and don't want to risk it happening to my baby. I'm confused on what to do. Because my husband said he never tried that again but I think the if I wanted to I would is disturbing.