Ghosting a parent

Not sure if this is right group or not but this made the most sense. So long story short, my dad is toxic. He's been a drug addict since I was 6 and has recently gotten clean with the help of binge drinking. So now he's an alcoholic that does illegal drugs on the side. He has a son that is 12 years younger than I am (7 about to be 8 in June) who is not in school and is constantly displaying horrible, disruptive behaviors. The other night when we were out to eat after not seeing each other for 6 months (he lives 7 hours away since moving to avoid paying a drug debt) my brother looks at me and says he hopes I lose my baby because he doesn't want it to ever be born. My dad looks at him and does nothing. If my dad or step mom isn't within eye shot of him he screams and panics and falls to the floor sobbing for them. My dad enables him which makes me sad for my brother. He makes very good money but quickly blows it all on drugs and alcohol then calls me and my husband for money for bills and rent. Anytime we say no he cries and yells and starts in on a guilt trip about how he's alone and no one helps him or he says he owns me and I have no choice. He's emptied my personal savings account four times, borrowed over $2100 from me, and over $500 from my husband while we were still dating. He's hit me, thrown me on the ground, hit me in the face with his belt, and gotten the cops called just to blame me saying I hit him (luckily my bruises showed them otherwise but he still got in no trouble). The point of all this is I want him out of my life. Him and his mother and his whole family. The only problem is the feeling like I'm doing something wrong. Like a little bit of guilt niggling at the back of my mind. And not knowing how I should go about it. Any advice is appreciated.