Daughter doesn't want to see her Dad

So some back story my daughter is 6 and her father and I have not been together since she was 3 weeks old. We have never involved the courts so I have full legal and physical custody and he does not pay child support. Despite this we have always had a schedule which since September has been 1 night one week and 2 nights the next. Our schedules have always been consistent.

Last October he got evicted from his rental house because of having to pay child support for his 1 year old daughter. He moved into his sister's basement which is just one big open room. My daughter started having some behavior issues and throwing fits every time we would drop her off there. So much so that I felt it was traumatic for her and debated on stopping overnights.For one, she didnt have her own space and two, he never brought her to do anything. They always just watched TV and she played on her tablet. Well two weeks ago he was supposed to move into a friend's house but the friend backed out the day of the move. His sister said he wasn't allowed back at the house as he was taking advantage. Therefore he became homeless.

Right away I told him he was not allowed to have our daughter overnight anymore but I wasn't going to keep her from him. Last weekend they were going to the mall and I said to drop her off at 2. I called and said we had some errands to run so he could drop her off at 3. He told me it was his time and he was keeping her and bringing her back when he wanted. I have never had to use the full custody card but I did that day and threatened to call the cops if he did not being her back. I am terrified he is going to just take her and not bring her back.

So since last October my daughter hasn't wanted to see him and only wants to stay with my husband and I. She doesn't give much of a reason anytime we ask. Her Dad offered to take her out for ice cream tomorrow after school for the last day and she doesnt want to go. I have told him that she doesnt want to go with him and he has seen her run and hide from him but still wants to force her to go with him. My daughter doesnt want to tell him she doesnt want to spend time with him because she doesnt want to hurt his feelings but she wants me to tell him. I've had this conversation with him before but he thinks it's my fault. He wants to blame him being homeless on "bad luck" not his bad decisions. He never takes the blame for anything and now he is putting the blame on me for my daughter not wanting to see him. I need opinions. He makes me feel bad anytime he has a short weekend or anything like that. I know it is important for kids dads to be in their lives but he has infuriated me so many times recently with all of this and I have put up with a lot of crap over the last 6 years. My daughter is old enough to voice her strong opinions. Do I stop their visits? My husband of course says I need to stop giving him chances but for some reason I still feel bad taking away his time with her even though my daughter doesnt want to go with him.