Am I wrong?

My mom and I have a rough past. The past 6 months she has not seen my daughter who is 18 months. She used to see her everyday and take her every other weekend. But I just couldn't get over the fact that she was abusive when I was little. She also lied a lot. But that's not the point. Her boyfriend has been around since I was 4, when my mom and dad got divorced. My mom has 7 grandchildren, who all call him grandpa. However, my 3 siblings see him as a father because they didn't have fathers, only him. I however did have my dad. I have never looked at him as my dad or even a father figure. I just don't feel anything towards him at all. The other day I went over to pick up some mail and he started yelling at me saying "you should be ashamed for keeping that baby away from her grandparents. She misses us, blah blah blah."

I respectfully told him that he is not her grandpa. He is not a part of my family. I just don't feel comfortable with that. They got mad saying that he is her grandpa, he was a father to me, that I can't choose that.

I simply just don't feel that way. I do keep my daughter away from them because my mom has done some horrible things and I told her I would be willing to talk about things if she would admit to everything and we go from there but I just don't think I want him around. He gives me a weird feeling. She told me no so we left and that was the end of it. She's now telling the whole family that I'm a bad mother for not wanting him to be the grandpa, and all this stuff.

Am I wrong for not wanting him to be there? I know everyone else sees him as family he just makes me feel weird and I'm not comfortable with him being considered her grandpa.

When I was 14 I was raped and my mom and him told him he was my stepdad so he was allowed in the room when I had to tell the story of what happened. He made me feel so weird and scared that I didn't tell them everything and my rapist got away. Everytime he's around I get this feeling like I need to leave. Even my husband feels weird around him. He's a great guy I just don't know why we feel that way.