Family issues

My baby daddy's parents want to take my baby away after he's born. They don't like me and don't want me to raise their grandchild. I've talked to a lawyer about it and he said they most likely wouldn't be able to get full custody and probably even no unsupervised visits before the baby stops breastfeeding. But I'm still scared they will be able to do something. Does anyone have experience with this?