Not pregnancy related

So I have a 2.5 year old with an ex, when he moved out, he moved to his mom's 4 hours away. Now at the time I was in rough shape, he was emotionally abusive and eventually left because I was catching on and went out and got a job. He left because he said I "abandoned him with the baby" while I went to work, I worked nights, sometimes had to stay till noon, and she slept through the night, had been for months before I got the job, woke up around 7:30-8am, she would be sleeping by 8 or 9pm.

Now because I was in rough shape emotionally after he was gone, I agreed to let our daughter stay with his mother for a bit, it started with a week, then a month, it kind of ended up with her being back and fourth every 2 months, until august, when I said it was just too much instability for her. Since then she's been down for just over a week at the end of November.

We'll now I don't want to send her down again. 1 he's not living with his mother anymore, moved in June or July. But he hasn't even asked about her in 4 months. While his mom talks to me often and asks to Skype sometimes. She's the one asking to take her. But she has a history of drug abuse.

I have reason to believe that her two sons who still live with her are both using coca in, her drug of choice. And also that they (not sure if she is included but clearly condoned it) smoke not only cigarettes but pot IN the house. The mother of her other granddaughter won't let her see her. And will only allow the father (my exes 14 year old brother) to see her in a public place every other week, if he bothers to see her at all. He apparently was physically abusive, and has anger issues resulting in acting out violently and I do not want my young child around that.

I don't live in the area, so I don't know first hand what goes on. But I believe what I've been told by this mother, as it fits what i already know about the family and even if I was sceptical about this information it's enough for me to say no i don't want my child there.

But I don't know how to tell the grand mother without telling her exactly why or who told me.

She's welcome to come here to see my daughter, so is her father. But I'm not comfortable with putting her in their care again.