Is This Wrong?

Kind of long but I need to rant and need some advice:
I just put in my two weeks notice at work yesterday as I have been offered an amazing job opportunity. The company I currently work for has been less than good to me, and I have been under a lot of anxiety for a while due to the way I've been used and mistreated. Needless to say, it's time to move on to something that makes me feel valued. Anyway, today I got a phone call from an angry co-worker who was one of my references. He told me that since he was a reference to me that now he's threatening to get fired. My boss is being extremely bitter towards me (who is my uncle) and is now threatening to fire my co worker just because he put in a good word for me as a reference. He isn't mad at me, he's just upset that he's on the verge of getting canned for nothing. I don't think this is okay at all. First, it's not a legitimate reason to fire anyone. Second, he didn't do anything wrong. Third, he did not confirm nor deny to my boss that I used him as a reference. It turns out my assistant "manager", who was not my boss as I am a manager too, is telling my boss that I'm sending inappropriate texts to this co-worker as a bribe for a good reference. He has been nothing but an ass kisser since I've known him and he's always been complaining about everyone to the boss. Because of me accepting a new opportunity, I'm now on the verge of just leaving without a replacement and my co-worker is about to get fired for no reason. Not to mention, since the boss is the owner and also my uncle, he's telling my family false information such as I'm committing payroll fraud, stealing credit cards, writing down hours I didn't work and more. What do I even do about this?! Do I just leave without hiring someone in my place or do I stand up for this?!
EDIT: forgot to mention that the "manager" found out that he was a reference through a text. One of my other co-workers eavesdropped on a private conversation and then told the manager that he was the reason I got my new job.....