I need advice!

Ashton
My SO and I were talking about the baby we're expecting the other day, and he asked what it would call his mom, and I told him whatever his mom wants. So then, he asked what it would call his moms boyfriend, and I said I don't really think he needs or should get a grandparent name, and my SO got really upset and said he needs to have one and that I'm being rude... 
a little background: His mom's boyfriend is physically abusive to her, and he is verbally abusive to everyone else. I have heard the phrases "brown people are taking our jobs", "you're going to hell because you don't go to church", and "being president is a mans job" come straight from his mouth. Needless to say, I have not one inkling of care for him, and neither does my SO, and I do not really want him around my daughter to either make her feel like she's worthless because she's a girl and because we aren't going to be raising her religiously, and for fear that he might hurt her physically if she makes him mad.
My question is, am I out of line for saying he shouldn't have a "grandparent" name and that I don't want him around my child because of all of this and the fact that he's just my MIL's boyfriend?