step dad and baby

Ok, I don't have kids and won't be trying for a couple years still but I need some advice. I have a very strong opinion about my step dad having nothing to do with my child/children. My husband thinks we should just bite our tongues when the time comes to keep the peace, but I do NOT want my step dad EVER touching my child. Growing up he was very mentally and emotionally abusive to me. I have so many ill feelings towards him, that thinking about him even touching my baby makes me cringe/want to puke. It's something I feel strongly about. So I guess I'm wondering if anyone else has had a similar situation and how you handled it? I don't want to piss everyone off and be shunned from my mom/sister but I just don't think I can do it any other way. I will add that my step dad is extremely childish and would throw a huge fit/freak out if I told him I didn't want him around my kids.. maybe I'm being childish? But when someone caused you so much emotional pain and abuse growing up, that still affects you to this day, i dont know how to not be childish about it..and he's just not someone I want my kids growing up around. His own daughter said she wouldn't care if he died, if that gives you any idea..