Advice appreciated!

Sorry for the long, possibly boring post but I am in need of some advice with hopefully someone who can relate. I'm a single mom to a 4 year old. She has somewhat of a relationship with her dad, who for the past years has taken her to his place for every other weekend or so. He also has a girlfriend and two other kids with her. 
My daughter is a sensitive child, just like me. She goes to daycare every week day and is socialble and enjoys going there. 2 weekends ago when my daughters dad came to pick her up, she ran into her room crying and saying that she didn't want to go. He got mad and just left while slamming the door behind him. I have kept her home on weekends since then because she always says that she "doesn't like going to daddy's" and she has told me that her younger sister (3 year old) hits her. I have brought this up with him and he just says that they play fine. She has also made comments about his girlfriend being mean sometimes. Anyways, he has blamed my parenting and told me that I coddle her and shouldn't be giving her so many hugs and kisses when she leaves. I don't know what to do because I want her to have a relationship with her dad but then again I don't want to be sending her to a toxic environment. Any advice?