Help please long thanks for reading.

So when my fiance and I moved in together and he started to bring his 2 kids over from another relationship which I would help him out but now it has gotten to the point that the mom couched the little girl to say that I spank her and that I bruised her face which she used all of this in court . And the little boy every time I call him out out something that he does he tells his dad that is not true making me look like a lier not only that the last thing that upset me is that I have a 2 week old baby girl and my baby was sleeping in the crib and he went to the room a yelled on her ear baby woke up got scared and cry . To all of this my fiance say Oh he is just a kid .and that the little girl would2fo nothing harmful bc she's a sweet girl . So last night he told me that it hurts him that I'm mean to his kids . I told him I wasn't mean all I did was back off and I don't do nothing with them bc first of all the mother of the kids lies so much so with her doing all the accusations against me in court I had told my fiance I didn't want to watch his kids no more bc I don't want problems so when they come i just stay in the room with my newborn and I let him deal with his kids .how can I tell him I don't hate his kids I just don't want to give my opinion in regards them I will help him to babysit when he's at work. that's all I don't want to hurt him