Don't want to visit abusive dad in hospital

My dad emotionally/verbally abused my mom, sisters, and I and on a couple occasions targeted my mom and one sister w/physical attacks. My mom never pressed charges but he did move out and continued to manipulate and harass us. Now he's recovering from major surgery he needed cause he hadn't been taking a necessary medication. I live a 2 hrs away and my family's (some know about the abuse, some dont) been pressuring me to visit him and saying I might regret it if things turn south suddenly. I do not want this man in my life till he makes an effort to change and I think it's more than enough that I text/call, and I accept that if he dies (and I hope he doesn't) I might feel guilt or regret. Am I being an asshole?