Creepy feeling - need opinions

My sister is married to a man who is very nice, yet gives me a weird feeling sometimes..

I have spoken to my husband about how I would prefer this man never be alone with our children, however I think he was a little taken back by my feelings. Its obviously not something I'd want to come out to anyone else in the family.. it could be very hurtful.

I guess what I'm asking is if you had a creepy feeling about a family member would you trust your gut?