I don't trust MIL's husband

I'm looking for advice on how to handle this stressful situation and so far I can't come up with a good solution. 
So my husband's parents are divorced. His mother remarried last year and I (along with the rest of the family) do not really care for the man. He's just a creep, to put it simply. He makes uncomfortable jokes, brings up odd topics of conversation and just gives off a very strange vibe that I can't stand to be around. I have been convinced he's a pedophile or the kind of guy who holds women captive in his basement but MIL really does love him and thinks he's great. So my husband and I suck it up and go over for uncomfortable dinners and holidays and be nice to him. 
Now I am 15weeks pregnant and I have a strong feeling that there's no way I could leave my child at their home to babysit or sleepover, like normal grandmas get to do. I just feel like this man is a bad influence and have a gut feeling that he may be capable of doing something inappropriate. My husband does not like this guy either but is angry that I'd deny his mom the ability to be a "normal grandma" and thinks she will hate me if I put down this limit. I said I'd be fine with her watching the baby at our house alone, but he knows she will want to do sleepovers and fun things at her house. Which I totally understand - some of my favorite memories are from when I went to my grandma's house when I was little. 
I know we have a few months before this is a real issue, but I'm not sure if I should talk with my MIL about it now, or should I wait to see if my feelings about creepy man change or just avoid the topic completely until she asks to take the baby to her house. I know she will be a good grandparent, I'd just prefer that the strange man not have unprotected access to my child. 
Can anyone offer any input on this?