Automatic Legal Writing Evaluation of LLMs

Abstract: Despite the recent advances in LLMs, benchmarks for evaluating legal writing remain scarce due to the inherent complexity of assessing open-ended responses in this domain. One of the key challenges in evaluating LLMs on domain-specific tasks is finding test datasets that are public, frequently updated, and contain comprehensive evaluation guidelines. The Brazilian Bar Examination meets these requirements. We introduce oab-bench, a benchmark comprising 105 questions across seven areas of law from recent editions of the exam. The benchmark includes comprehensive evaluation guidelines and reference materials used by human examiners to ensure consistent grading. We evaluate the performance of four LLMs on oab-bench, finding that Claude-3.5 Sonnet achieves the best results with an average score of 7.93 out of 10, passing all 21 exams. We also investigated whether LLMs can serve as reliable automated judges for evaluating legal writing. Our experiments show that frontier models like OpenAI's o1 achieve a strong correlation with human scores when evaluating approved exams, suggesting their potential as reliable automated evaluators despite the inherently subjective nature of legal writing assessment. The source code and the benchmark -- containing questions, evaluation guidelines, model-generated responses, and their respective automated evaluations -- are publicly available.

• The paper introduces oab-bench, a benchmark that uses LLMs to generate and evaluate legal exam responses, achieving notable alignment with human scoring.
• It employs a dual role design where LLMs act as both candidates and evaluators, leveraging comprehensive guidelines from the Brazilian Bar Examination.
• The study finds that Claude-3.5 Sonnet excels with an average score of 7.93, underscoring the potential of LLMs in automating subjective legal assessments.

Automatic Legal Writing Evaluation of LLMs

The paper "Automatic Legal Writing Evaluation of LLMs" (2504.21202) presents a novel benchmark, oab-bench, designed to evaluate the performance of LLMs in the legal domain, specifically focusing on the Brazilian Bar Examination. This work addresses the complexity of assessing open-ended responses, a task characteristic of the legal domain due to its subjective nature.

Introduction and Motivation

The challenges associated with the automated evaluation of legal writing are multifaceted, primarily due to the open-ended and interpretative nature of legal tasks. Prior benchmarks such as LegalBench have laid the groundwork by incorporating interdisciplinary collaboration to represent real-world legal practice tasks. However, these often require human evaluators. The introduction of oab-bench aims to fill this gap, offering a structured, comprehensive evaluation framework leveraging LLMs as automated judges.

Methodological Approach

Benchmark Design: oab-bench

The oab-bench includes data derived from the Brazilian Bar Examination, a national examination required for law graduates to practice in Brazil. This benchmark comprises 105 questions spanning seven legal areas, each accompanied by comprehensive evaluation guidelines, reference materials, and detailed score distribution tables. This design ensures rigorous, consistent evaluation aligned with human examiner protocols.

Dual Role of LLMs: Candidate and Judge

The study showcases two roles for LLMs: as candidates generating answers to exam questions and as judges evaluating these answers. The generative aspect leverages specific prompts to mimic the exam conditions faced by human candidates. As judges, LLMs are tasked with assigning scores based on detailed evaluation criteria, such as legal argumentation quality and alignment with provided guidelines.

Evaluation Pipeline

The evaluation pipeline employs frontier models like OpenAI's o1 to assess model-generated responses' alignment with human experts' scores. The results demonstrated a high correlation between LLM-assigned scores and human evaluations, suggesting LLMs' potential as automated evaluators of legal writing.

Results and Findings

The evaluations revealed that Claude-3.5 Sonnet outperformed other LLMs, achieving an average score of 7.93 across all exams and passing every test. The study indicates that LLMs can substantially align their scoring with human judges, achieving a notable correlation in their assessments. This result underscores LLMs' growing role in automating complex, subjective evaluation tasks within specialized domains.

Figure 1: Performance of each model across different areas of law.

Implications and Future Directions

The implications of this research extend towards the broader application of LLMs in automating subjective assessments in specialized fields like law. The demonstrated ability of LLMs to replicate human judgment in legal writing evaluations hints at significant cost and resource savings. Future developments could explore enhancing LLM evaluation accuracy through domain-specific fine-tuning and incorporating additional interpretative data.

Conclusion

The paper provides a significant contribution to the landscape of LLM evaluations in specialized, open-ended tasks. By introducing and validating the oab-bench, and exploring LLMs' dual roles as both candidates and evaluators, this work sets a precedent for further exploration of AI capabilities in complex, subjective assessment contexts, heralding advancements in automated legal writing evaluation.