When to start weekly stays with ex (my son, not me lol)

My Son is not quite 6 months old yet and my ex wants to start taking our son every other week.. He has this same routine with his ex wife but his daughter is 4 and half the time she's with her dad his 85 year old grandmother is watching her while he works (which is 1-9 every day of the week).. I'm scared that something will happen to his grandmother while she's watching my son and his daughter and no one will know and they'll be left alone.. Also he does not watch his daughter like he should and she gets into stuff often when he's supposed to be watching her.. I don't know what to do.. Any one deal with this as well and if so when did you start week on week off stays?

Lisa she does a lot more than just watch his daughter. She washes his clothes, makes his lunch for work, and cleans his house.. It's been like that since I met him 2 years ago. He's not responsible. He says she wants to do it but why would an 85 year old woman want to take care of her grown ass grandson??