Issues with father?

So long story short, my parents were married when I was growing up and when I was 13 they got a divorce. I'm now 22 and since the divorce it seems like my dad has been intentionally trying to avoid my mom, my siblings, and myself. He only lives about 5 minutes away from us and never calls or visits. My mom says that he can come visit whenever he wants to but he doesn't make any afford. Today he decided to stop by and see us and didn't interact with us at all. Mind you my little brother is special needs and my dad seems to resent him the most. He was trying to play with my dad by pulling on his shirt and touching his face and all my dad could do is yell at him so my mom told him to leave and never come back. It took me forever to get over the fact that my dad doesn't see us and today after that whole thing it made me feel super depressed. I feel torn because part of me wants absolutely nothing to do with him and then the other half still wants a relationship with him. I'm just venting here but does anyone have any input? It would be greatly appreciated ❤️