Baby sitting a troubled kid

So I have recently started baby sitting a boy almost 6 years old and I have an almost 5 year old. When interviewing with his dad he explained to me that his son had a very unstable life for the first 4 years. His mom being a drug addict and him not getting custody til he was three but even then things were unstable due to trying to get in his feet. 
He said he had anger problems and had been told he did things like spitting and hitting but said that only that last sitter said that and no one else ever had. I came from a life similar so I understand how hard it is. I told him I can handle it, and I can. I have a ton of patience. 
But here is my concern now, the bad behavior is affecting my daughter. NEVER would she dare to laugh or smile while being scolded but he does and now it's happening with her. She it trying to pull a lot of things he does and I'm not okay with this from her. She gets in trouble every time and he does as well but he doesn't listen for anything! Nothing I can do makes him listen. It's all a joke and a game to him. But fine, you want to spend all day in time out then go home and get in trouble with your dad, so be it. I'm concerned about the negative affects it's having in mine. 
We are very big on feelings and discussing things. She has always been so good and so open and in control of herself til now. 
Would you continue to watch him or would you stop because of the way it was affecting your kid?