Custody battle

(Don't know where to post this.)
The father of my son and I broke up this past Friday and I moved in with my brother and his family. My ex was mentally and emotionally abusive and would have violent outbursts where he would punch holes in the wall and break things. This even happened once while he was caring for our son. The second fight we had was nasty and his mother was present and she said something about how she would file for custody of my son. I haven't been returning any of her phone calls or texts since then. Today she said that if I don't answer her she will "take things into her own hands" which I'm assuming is try and get custody. I'm wondering how possible that is. I've only just returned to work, working only weekend nights. I don't have a car at the moment so it's been difficult getting back and forth but I've managed. My son is 5 months old, he eats only breastmilk and I'm basically all he knows. His father has been less than present, though we were living together. He's an alcoholic and smokes weed heavily. He never made time for myself or my son, instead choosing to play video games. This is not the first time my son's grandmother has threatened me in some way. She's pretty crazy and, for lack of a better word, scummy. And her husband is basically a worst version of his son. How likely are they to get my son? What would deem me unfit?