From Citations to Criticality: Predicting Legal Decision Influence in the Multilingual Swiss Jurisprudence

Abstract: Many court systems are overwhelmed all over the world, leading to huge backlogs of pending cases. Effective triage systems, like those in emergency rooms, could ensure proper prioritization of open cases, optimizing time and resource allocation in the court system. In this work, we introduce the Criticality Prediction dataset, a novel resource for evaluating case prioritization. Our dataset features a two-tier labeling system: (1) the binary LD-Label, identifying cases published as Leading Decisions (LD), and (2) the more granular Citation-Label, ranking cases by their citation frequency and recency, allowing for a more nuanced evaluation. Unlike existing approaches that rely on resource-intensive manual annotations, we algorithmically derive labels leading to a much larger dataset than otherwise possible. We evaluate several multilingual models, including both smaller fine-tuned models and LLMs in a zero-shot setting. Our results show that the fine-tuned models consistently outperform their larger counterparts, thanks to our large training set. Our results highlight that for highly domain-specific tasks like ours, large training sets are still valuable.

• The paper introduces a dual-labeling system that assesses case criticality by both citation frequency and timing.
• It employs a semi-automated annotation method with fine-tuned multilingual models that outperform zero-shot approaches.
• The study highlights the impact of language-specific variations, emphasizing the need for task-specific NLP model adaptations.

Breaking the Manual Annotation Bottleneck: Creating a Comprehensive Legal Case Criticality Dataset through Semi-Automated Labeling

This paper presents an innovative approach to addressing the significant challenge in the legal domain of predicting case criticality through the development of the Criticality Prediction dataset. By employing semi-automated labeling, the authors offer a resource-efficient alternative to manual annotations, enabling a more extensive dataset.

Summary of Contributions

The research introduces the Criticality Prediction task, aimed at forecasting the influence of Swiss Federal Supreme Court decisions on future legal precedents. The dataset is characterized by a two-tier labeling system: the LD-Label, classifying cases as Leading Decisions (LD), and the Citation-Label, which evaluates cases based on citation frequency and timing. This dual system provides a nuanced perspective, distinguishing cases not only by criticality but also by their temporal impact.

In the evaluation stage, various multilingual models were tested, both fine-tuned and in a zero-shot capacity. Fine-tuned models consistently outperformed zero-shot baselines, underlining the importance of model adaptation to specific tasks within legal NLP.

Key Results and Findings

1. Dataset Characteristics: The dataset includes cases from the Swiss Federal Supreme Court, annotated using a semi-automated approach. The LD-Label is binary while the Citation-Label ranks cases into four levels of criticality, considering both citation frequency and recency.
2. Model Evaluations: Multiple multilingual models were evaluated, including well-known architectures like XLM-R, mDeBERTa, and SwissBERT. The models were assessed in scenarios using different languages and input types (facts and considerations). The results demonstrated that fine-tuning provided a significant advantage over zero-shot approaches in handling the dataset's complexities.
3. Language and Input Variability: The study explored the impact of language-specific variables, revealing that performance varied across German, French, and Italian datapoints. This insight highlights the inherent linguistic diversity encountered in such multilingual legal datasets.

Implications and Future Directions

The introduction of this dataset and the Criticality Prediction task represents a significant step toward automating legal document analysis, offering both theoretical and practical advantages. Practically, this could streamline processes in the legal field by aiding in the prioritization and assessment of case importance, influencing how legal resources are allocated. Theoretically, the dataset opens new avenues for research in legal NLP, specifically for those working with case law in multilingual contexts.

Future research may explore expanding this framework to other jurisdictions, offering comparative insights across different legal systems. Furthermore, integrating this approach with more advanced models could refine its applicability and accuracy, enhancing its utility in real-world legal settings.

In conclusion, this paper makes substantial advancements in legal NLP by tackling the annotation bottleneck, providing a detailed, scalable dataset, and demonstrating the need for task-specific adaptations in predictive models. This work sets the stage for further exploration and application of artificial intelligence in the legal domain.