I don’t trust my partners dad with my kids

my partner trusts his dad but I don’t as family friends of theirs cut ties with them as 2 of their sons 12 and 9 stated that my partners dad was touching them inappropriately my partner said they were lies and so did his siblings but I just found it odd because he would take the boys out bowling and to eat and not take his own kids and his dad also takes his 14 year old nephew out and doesn’t invite anyone else not even his 16 year old daughter and I just find it odd and something in my gut tells me something isn’t right about the man he normally doesn’t talk and is selfish to his children and plays video games when he’s not working he’s building lego but something just tells me not to trust him and I don’t wanna tell my partner I feel that way as I know it could hurt him so instead I just don’t allow our kids to go over unless I’m with them but he asks to take the kids out to concerts and activities and I’m running out of excuses