Should I make him choose?

When I was 14, I was raped by my brother. Half brother actually. I'm now 19, so it's been a while. When my mom found out she called my dad and he told her not to go to the police. She did anyways and a year later, he was arrested. He served about 6 months but was sentenced to 4 years. My dad, and his side of the family sent him money while he was in there, quite a bit too. They called him, went to visit him. For that year, my dad and I were very very distant. I didn't want to talk to him, it was weird. Eventually I got over it because my mom really wasn't there much so he was all I had and he was always my best friend. After about 2-3 years my dad would talk about him a lot. "Me and ___ are going camping" "I'm going over to ___ house to watch the game" "his kids are coming over" etc. it's his first born son. Lately it's really been hurting me. They recently went camping.. the entire family. Mind you, I haven't seen my other brother or sister in three years because they live out of state. They all drove 10 hours to go camping. He was invited.. I wasn't. When Christmas comes, they will have two days. One he will come, one I will. On Easter, my daughter missed out because he was invited. My dads girlfriend said she tries to make it so we both are able to come but.. why should he? He's a rapist!!!! I love my dad, I do. He doesn't handle conflict, at all. He just ignores it. I've never told him how I felt. I'm wondering if I should just stop talking to him or if I should tell him it's either me or him? It's his son.. so can I even do that? But then again.. why do I have to suffer because of him? Why do I get to know that he's living great? He just had a baby, bought a house, got a raise at his job (which by the way they HELD FOR HIM WHILE HE WAS IN JAIL). He fucked up my childhood. I dropped out of school, became depressed, tried to kill myself. Yet he's the one that gets invited to everything.

Last year we went to my aunts Christmas and EVERYONE was complaining because he wasn't there, not one person even said hi to me. When we left, no one noticed. I feel like it's either he chooses or I leave. But I'm not sure..