Am I wrong for not letting my son go to his dad's for Thanksgiving

My kids are 12 and 14 and have different dad's. I won't pretend I was a good girlfriend to either of these men but I cheated on my 14 year old daughters dad and she still hasn't tried to make my life hell like my 12 year old sons father. Like I said I wasn't a good girlfriend to either of them. My daughter's dad I cheated on and my son's dad I more or less played around with his feelings and left him. Not an excuse but I was 20 then and very young. I had kids young. I was a mess. I know his dad hates me but honestly when will I have finally suffered enough for him to leave me alone. When me and his dad broke up my son was around 18-20 months. I know for a fact he was under 2. His dad hates me and I understand but he has always wanted full custody of our son. Not because I'm unfit but because HE wants me completely out of his life and doesn't wanna coparent. He's made co-parenting hell. We have split custody and the worst thing he did to me was I asked if I could have my son with me to go to Disneyland with my parents when my son was 6. It would be on his custody day and I would accept a no. He said yes though and was completely fine with it. While we were on the trip he called the police and said I broke the custody agreement and KIDNAPPED our son! He put me through HELL with that because everything said was a verbal agreement so I had no proof he gave me permission and ended up in contempt court. He has tried over the years to trick into violating our custody agreement and I try to only communicate to him through lawyers and I get EVERYTHING I'm writing now. His family has harassed me and I'm so paranoid that I don't accept anything from him because I'm terrified he's trying to plant drugs on me and something to make me seem as am unfit mom. He actually had to miss his holiday with our son Because he got hurt but I had that in writing and had the conversation between our lawyers so he couldn't say I kidnapped our son. He messaged me saying that since he didn't get our son last holiday can he have him for Thanksgiving, even for a few hours. I said no because it's not his custody day. He called me selfish and said he knows I've done that for my daughters dad. My daughter's dad hasn't been trying to spend the last 6 years trying to RUIN me and take my custody away. He made me feel awful and then his family said what they had to say. It sucks because I get it. I wasn't a good girlfriend to him. But it's not like I abused him or did anything to prove I'm a violent person that would warrant my son being taken from me. When will I have suffered enough in his eyes? When will he have put me through enough for him to be satisfied? My son is 12 and I know he loves us both and I want him to have a good relationship with his dad. It just sucks that his dad wants to alienate me from his life. The right thing to do would have been let him see his son on Thanksgiving but I just don't trust him

@Realist it's not that simple. We have a custody agreement and I wouldn't be able to just break if my son wants me too. If I could trust his dad it would be fine but this man has tried to get my son taken away and tried to trick me into breaking the agreement.

Edit: I just want to say that had it been through our lawyers I would have said yes. He had all this time to aske THROUGH OUR LAWYERS and he waited until the day of on a Holiday so I couldn't and in the past he's tried to get me to give our son on a custody day that wasn't his and i found out he had plans to report me for abandonment. It's why everything is through our lawyers. He also had the option to come to my house to see our son and he said no. Had he not been trying to take my son from me for 6 years then we wouldn't be in this situation.