Why are so many people worried about what others do?

If it doesn't directly affect you why care?