Coparenting is really fucking with my mental health

I feel like I’m all on my own. My sons father chooses when he wants to act like a dad. He constantly disrespects me when I mention how absent he is in my sons life. When he does come to visit and take my boy, my little one acts distant and doesn’t let himself be hugged by him. It breaks my heart to see how it’s affecting him having a father in & out of his life. I get blamed for all this because I have one rule and it’s that his mother can’t be near my kid. My child used to come back from her house sick all the time and even came back with a huge bruise on his forehead once. She’s harassed, threatened, stalked, and tried getting physical with me so I had to make a police report and from then on, contact was cut off 100%. He says if I allowed him to let my boy go to their house he wouldn’t be so absent. I keep trying to stay in contact with him and try to come to an agreement but I always feel so drained and depressed after I speak to him about our son. Is this my fault?