Am I wrong for not seeing my uncle in prison?

li
My uncle went to prison for beating up a crack head. Literally. He went to prison for 20 years. 
When this all went down my whole family was so upset, everyone wanted to be in his defense. But I knew that if he wasn't doing the things he was doing like being involved in drugs and drug dealing he wouldn't be where he is. My whole life I wondered how he did what he didn't and never got in trouble. He's in a low security prison because he had cancer and is still in remission. My family goes up there every Saturday and I don't, we sit in a room with 50 other inmates in their families and they make me highly uncomfortable. I've been before and he's told
Me "we are sitting next to a man who raped a 6 month old then killed her" 
WHAT THE FUCK I DO NOT WANT TO BE AROUND THOSE PEOPLE. I'm sorry but you landed yourself there and it's not my job to revolve my life around seeing you in prison. He only has like 5 years left due to court appeals and all that, I'm worried he'll be mad at me when he comes out. Am I in the wrong? Should I be going to see him every weekend?