5yr old refusing to see dad

My sons dad and I got divorced when he was 2 and stopped living together at 1. My son has always lived with me and been a happy little guy except he’s never been happy about visits.

Recently he said he doesn’t want him anymore because he remembers him hurting mommy, he’s mean and gross. There’s a HUGE showdown every time it’s time to go… like screaming, crying, running away and at this point it’s just heartbreaking because legally I can’t do anything unless he’s abusing him. He has been asking him to stop coming but dad thinks my son is manipulative and just being a baby… he’s also thinks he’s autistic even though he’s been evaluated and it hasn’t come up and that’s what it is. I told him my sons concerns and he blows them all off. He’s in a horrible situation financially and recently shared his gf is having a baby soon so my sons going to be worst off than he already is there. She’s overbearing and now he doesn’t want to see her either which makes it worst. I’ve done everything to make him feel better about going but he says it makes him sad when we don’t listen to him and he has to go. At this point idk what to do. I wish things were better with his dad and he was happy to go but the older he gets the more he refuses and fights.

Has anyone been in a similar situation and have advice? I just want my child to be happy