custody issues

I have primary custody of my kids and my ex is supposed to take them every other weekend and pay child support. He hasn't paid for years and we are finally going to court for it next month. He has a history of addiction and cycles between using and recovery more times than I can count. His visits have had some negative events happen, relatively minor but enough that I was concerned about their safety, plus several altercations when I went to pick them up. He also abused me in every way during our relationship and having to Co parent with him is almost always a nightmare. We have been divorced for almost 6 years. He usually doesn't even ask for his visitation, and has gone months with no contact. the past year, he is living with a woman with kids and she seems ok. I have been arranging his visits at his mothers house for the kids safety, and told him that I would let the kids stay at his house when he has been seeing and talking to the kids regularly, and not just one visit or two and then no contact for a month or longer. He will ask to see them about once a month. Now that he was served for court for not paying child support, he has his copy of our court agreement that he lost years ago and threw a fit about how I keep the kids from him and that he has rights. which is true, but he almost never exercises those rights! Anyway, I never took him to court to change the custody because I can't afford a lawyer, and he usually agreed to the changes I made (that his visits be at his mother's house). His mother doesn't have him over for these visits sometimes because she tells me "it's like having another child" and that she is glad her husband is there with her so she wouldn't be alone with her own son (she has to walk on eggshells because he will pick fights if he's in the wrong mood). I agreed to send the kids to his house with his gf who is also a mother, for a visit to see how things go. I don't want to get in trouble for effectively withholding the kids (although he has not once shown up to get them like the court papers specify. he's supposed to pick them up or arrange transport to his house and he has not once tried to), but I'm also nervous that I'm making a mistake and my kids could get hurt somehow because I don't trust his judgment. I don't know if it's the right thing to do to give him another chance for the 20th time. or if I'm being unreasonable because while his house or parenting isn't up to my standards, he is their dad and he seems to be doing better for himself for the past few months. What would you do?