Mom and her boyfriend

I have a 1 year old daughter and him and my mom just decided that he's her grandpa, I don't feel comfortable with that. He's not my dad. 

He will just come over and pick her up whenever he wants and again, I don't like it. 

We live with my mother so we don't have a say otherwise we get kicked out. My mom just will come up and grab her from me and take her downstairs to meet people I don't know. She freaks out when I don't let her take her to see my grandma because she's gone for hours and always takes her other places. I have told her I will take her to see grandma and she tries to guilt me. 

I've tried to talk to her but it's to the point where I'm about to tell my husband I just want to leave and I don't care where we go. 

Growing up my mother was extremely abusive. She threw us down stairs, kicked my brother in the face until he bled, threatened to kill us, told me I ruined her life, etc. 

She's never apologized and while she's never done anything to my baby, I'm terrified she will. However if I bring that up, she will call everyone in the family and lie and say I'm making things up about her. I can't do this anymore. My dad says I can't just leave and never let her around but honestly, part of me wants her out of my life.

We broke the news to her the other day that I have ovarian cancer. Her response? Oh that's not too bad, just get a hysterectomy. I tried to explain to her what the doctor said and she just kept saying "oh no your wrong. It's not dangerous. You'll be fine." Yet 80% that have it die. 

Any advice?

Everyone's saying move out. We can't. We will be homeless.