Am I wrong?

My children father has been asking to get them but I am afraid for them to go to his house. I would prefer him coming to see them, seeing them at his mom's house, or taking them to a public place supervised. The reason for that is because he started selling drugs and he sells them either out of his house or goes to the people house to give it to them. I know he would go around with my children in the car distributing drugs or have the people come to his house to buy them. I am an anxious mother and I try my best to protect my children. He didn't start doing thing until after the kids were born. He also stays in and out of jail plus there is a protective order in place where he can't be around me for 6 months due to him threatening me. I want him to see the kids but I afraid. Am I wrong for not wanting him to get the kids or should I just let them go and stop worrying.