Prompt-Chaining Approach

• Prompt-chaining is a methodology that breaks down complex LLM tasks into sequential, specialized sub-tasks to improve controllability, transparency, and accuracy.
• It utilizes modular steps such as summarization, exemplar retrieval, and few-shot generation to effectively manage long texts and domain-specific challenges.
• Empirical results demonstrate significant improvements in metrics like micro-F1 and Exact Match in legal tasks, validating its practical impact.

Prompt-chaining is a methodology for decomposing complex language-model tasks into structured sequences of discrete LLM invocations, where the output of each stage is fed as input to the subsequent stage. The primary rationale is to orchestrate sub-task specialization, error isolation, and explicit intermediate representation, yielding gains in controllability, transparency, and accuracy—particularly when input sequences are long or domain complexity is high. In legal document analysis, prompt-chaining pipelines can overcome fundamental bottlenecks of context window limits, information overload, and domain-specific language, making them especially well-suited for tasks such as long-text classification, contract QA, and similar high-stakes applications (Trautmann, 2023, Roegiest et al., 2024).

1. Architectural Principles and Formal Definition

A prompt-chaining pipeline is a composition of modular LLM calls, each encapsulated as a function $f_i$, often with distinct prompt templates and domain-adapted constraints. Formally, if $x$ is the original input (e.g., full document), the chain is

$y_{n} = f_{n}( \dots f_2( f_1(x) ) \dots )$

with each $f_i$ potentially depending on intermediate outputs $y_{i-1}, y_{i-2}, ...$ as required.

In long legal document classification (Trautmann, 2023), the canonical three-stage chain consists of:

1. Summarization ($f_1$): Compress and distill input $x$ to a 128-token, task-relevant summary using a domain-tuned abstractive model (e.g., PRIMERA).
2. Exemplar Retrieval ($f_2$): Retrieve $k$ labeled in-context exemplars by computing cosine similarity in an embedding space (e.g., LegalBERT sentence-transformer).
3. Few-shot Generation ($f_3$): Construct a prompt that concatenates the $k$ nearest (summary, label) pairs and requests a label prediction for the new summary via LLM decoding.

For contract QA (Roegiest et al., 2024), a two-stage chain is employed:

• Stage 1 ($f_1$): Extract only those facts in the clause relevant to the posed question (“distillation”).
• Stage 2 ($f_2$): Map the distilled intermediate to a discrete set of options.

2. Prompt-Chaining Workflows in Legal Domains

The prompt-chaining approach enables efficient processing of documents far exceeding standard transformer windows (e.g., ECHR, SCOTUS opinions spanning 2048+ tokens):

Document Summarization:

• Input is iteratively segmented into context-window-compatible spans (1,024–2,048 tokens).
• Each span is summarized using PRIMERA (fine-tuned on legal opinion data).
• Sequence summaries are recursively condensed until a single global summary of 128 tokens remains.
• Default LLMs (e.g., GPT-NeoX, Flan-UL2) using generic summary prompts were found to dilute high-value legal signals; PRIMERA summaries retained legal issue structure and proved empirically superior (Trautmann, 2023).

Semantic-Similarity Retrieval:

• All documents (including test, train, dev) are pre-summarized.
• Each summary is embedded; for a given test query, $s(q, e) = \cos(\text{Emb}(q), \text{Emb}(e))$ is computed against training samples, yielding the top $k=8$ most similar exemplars.
• Exemplar (summary, label) pairs serve as explicit in-context learning signals for the subsequent prompt.

Few-Shot Label Prompting:

• Final prompt concatenates 8 retrieved pairs in descending similarity.
• For binary tasks (ECHR): label set is {YES, NO}; for multi-class (SCOTUS): restricted to labels present among nearest neighbors.
• Greedy decoding ($T=0$) or majority voting over $n$ samples (self-consistency) is used for the final label.

Contract QA Chaining:

• Stage 1 prompts force the LLM to report only explicit, question-relevant facts—no inference or assumption.
• Stage 2 receives this distilled content and applies structured option selection, outputting selected answers in JSON.

3. Empirical Evaluation and Performance Metrics

Classification Task Results (Trautmann, 2023):

ECHR (binary):

• Zero-shot GPT-NeoX (full doc): micro-F₁ = 0.709 (dev), 0.728 (test)
• Chained GPT-NeoX (8-shot, summaries): 0.770 (dev; +0.061), 0.756 (test; +0.028)
• NO-class F₁ improved from ≈ 0.15 (zero-shot) to 0.25 (few-shot)

SCOTUS (13-way):

• Zero-shot ChatGPT: micro-F₁ ≈ 0.438
• Flan-UL2 (chained, 8-shot): micro-F₁ = 0.545 (dev), 0.483 (test; +0.045 over ChatGPT)

Micro-F₁ improvements over strong zero-shot and single-prompt baselines are consistent, despite using smaller models.

Contract QA (Roegiest et al., 2024):

Summary of Prompt Chaining vs. single-stage:

Prompt	Q1 EM	Q3 EM	Q4 EM
P1	0.47	0.51	0.68
P3	0.51	0.51	0.54
P4	0.57	0.58	0.68

Chaining (P4) yields +6–7 point EM gains on complex questions.

Metrics

• Micro-F₁ and Macro-F₁ for classification
• Exact Match (EM), Precision, Recall for multi-select QA
• Self-consistency: negligible gains over greedy decoding, suggesting improvements are due to exemplar context, not majority voting.

4. Design Patterns, Component Modularity, and Practical Considerations

Modularity:

• Each pipeline step is swappable. For example, summarizer, embedder, k-neighbor count, or decision rule can be replaced without affecting downstream structure.
• In contract QA, Stage 1 can be realized as summarization, slot-filling, or explicit right/obligation extraction; Stage 2 as any structured answer selection.

Prompt Engineering:

• Task-specific summaries outperform generic LLM-generated ones; human review confirms greater retention of legal issues.
• Including answer options in Stage 1 summaries can guide focus but risks “pocket prompting” leakage—favors experiment-driven tuning.

Limitations:

• Chaining exposes error propagation: poor Stage 1 outputs limit final answer quality.
• Linguistic variability (e.g., in force majeure scenarios) still leads to failures; neither exhaustive prompt definitions nor chaining fully solve high-variation semantics.

5. Theoretical and Methodological Rationale

Prompt chaining, by construction, enables fine-grained control over task decomposition and input filtering:

• Error Isolation: Partitioning allows upstream errors to be detected and corrected before contaminating final decisions.
• Context Compression: Summarization circumvents transformer context window bottlenecks.
• Few-shot Efficiency: Chained few-shot prompts yield performance matching or exceeding larger model zero-shot runs, reflecting improved in-context signal (Trautmann, 2023).

In contract QA, chaining aligns with the expert process: extract relevant facts, then reason/disambiguate on the distilled set (Roegiest et al., 2024).

6. Broader Implications and Future Work

The modular chaining paradigm positions each sub-task for focused error analysis and component-wise optimization. Future research avenues include:

• Task-specific intermediate extraction (beyond summarization) such as party rights/obligations, structured slot filling, or intent detection (Roegiest et al., 2024).
• Robustness across LLM versions (GPT-4, Llama-2, Mistral).
• Chaining more than two stages, with justification-explanation decoupling (e.g., requiring rationales in option selection).
• Model-level enhancements: pretraining on chains of contract QA pairs, data augmentation for high-variation phenomena.
• Addressing overfitting to known answer structures by prompt randomization and cross-exemplar testing.

Taken together, prompt-chaining delivers a scalable, transparent, and empirically validated approach for handling long, complex, and structured language modeling tasks in legal and other high-precision domains, systematically outperforming single-stage prompting and larger zero-shot LLMs (Trautmann, 2023, Roegiest et al., 2024).