Generative Information Extraction

• Generative Information Extraction (GIE) is a paradigm that reformulates extraction tasks as conditional text generation, unifying NER, RE, and EE under one model.
• It leverages large language models with techniques like supervised fine-tuning, few-shot learning, and prompt optimization to produce structured outputs such as JSON or linearized schemas.
• GIE advances traditional pipelines by robustly handling complex, overlapping, and noisy inputs across diverse domains, thereby enhancing scalability and accuracy.

Generative Information Extraction (GIE) is an information extraction paradigm that formulates the derivation of structured knowledge from unstructured text as a conditional text generation task, typically implemented using LLMs in an encoder–decoder or decoder-only architecture. This approach unifies diverse IE subtasks—including named entity recognition, relation extraction, and event extraction—under an auto-regressive generation objective, allowing a single model architecture to produce heterogeneous structured outputs from natural language input across a variety of domains (Xu et al., 2023).

1. Formal Paradigm and Mathematical Foundation

At its core, GIE models the probability of generating a structured output sequence $Y = [y_1, …, y_m]$ from an input text sequence $X = [x_1, …, x_n]$ and an optional prompt or instruction $P$. Model parameters $\theta$ are typically initialized from a pre-trained LLM and then adapted via supervised or semi-supervised fine-tuning. The generative formulation is defined as:

$$p_\theta(Y | X, P) = \prod_{i=1}^m p_\theta (y_i \mid X, P, y_{<i})$$

where $Y$ encodes the required structured outputs—entities, relations, events—in a specific linearized schema (e.g., plain text, serialized code, JSON). The training objective is maximum likelihood estimation, minimizing the negative log-likelihood:

$$L(\theta) = - \sum_{(X,P,Y)\in D} \sum_{i=1}^m \log p_\theta (y_i \mid X, P, y_{<i})$$

This generalizes classical sequence labeling to flexible, arbitrarily structured output spaces (Xu et al., 2023).

2. Taxonomy of Generative IE Subtasks

GIE systems are applied across a variety of information extraction subtasks, each mapped to a specific generative schema:

• Named Entity Recognition (NER):
  • Entity Identification: Output spans $S = \{(s_j, e_j)\}$.
  • Entity Typing: Assigning type $t_j \in T$ to each $s_j$.
• Relation Extraction (RE):
  • Relation Classification: Predicting $r \in R$ for a pair of spans $(h, t)$.
  • Relation Triplet Extraction: Generating all $(h, t, r)$ triples.
  • Relation Strict: Recovering head and tail entity types alongside the relation.
• Event Extraction (EE):
  • Trigger Detection: Detect and classify event triggers $\tau_j$.
  • Argument Extraction: For each $\tau_j$, output role–entity pairs $(\tau_j, a_k, \mathrm{role}_k)$.
• Universal IE: Recasting all tasks in a unified schema:
  • NL-LLM-based: Structured predictions are expressed in natural-language templates.
  • Code-LLM-based: Outputs are serialized as Python or other code structures encoding the schema (Xu et al., 2023, Fei et al., 2023).

3. LLM-Based GIE Techniques

GIE leverages the modeling capacity and transferability of LLMs along several adaptation axes:

• Supervised Fine-Tuning: Full or partial updating of parameters $\theta$ using labeled datasets. Parameter-efficient approaches, such as prefix-tuning or LoRA, inject additional adaptive matrices while freezing the backbone. For example, LoRA updates the hidden states $h_\ell \gets h_\ell + A_\ell B_\ell h_\ell$ (Xu et al., 2023, Choi et al., 20 Apr 2025).
• Few-Shot Learning:
  • Fine-Tuning: Training on $k \ll |D|$ examples; methods such as TANL and CP-NER optimize task-specific prefixes.
  • In-Context Learning (ICL): Concatenating a small set of annotated demonstration pairs with the target, keeping model weights fixed (Xu et al., 2023, Cao et al., 2023).
• Zero-Shot Prompting: Designing $P$ so that the LLM can perform extraction without any $X$–$Y$ pairs. Examples include recasting tasks as question answering (QA4RE) or using chain-of-thought (CoT) prompts with intermediate reasoning steps (Xu et al., 2023).
• Instruction Tuning & Chain-of-Thought: Training or prompting models with explicit instructions or schema guidelines, and optionally inserting explicit reasoning steps between context and answer (Xu et al., 2023).
• Data Augmentation: Synthetic data is generated via prompting the LLM to create annotation candidates, inverse text–structure pairs, or by leveraging the model itself as a weak annotator (Xu et al., 2023).
• Multimodal Adaptations: In domains such as scanned or visually formatted documents, models (e.g., GenKIE, GMN, Generative Compositor) fuse text, layout, and visual features in the encoder via 2D positional embeddings and convolutional layers, generating directly from modality-fused representations (Cao et al., 2023, Cao et al., 2022, Yang et al., 21 Mar 2025).

4. Output Schematization and Prompting

A central aspect of GIE is the explicit definition or linearization of the output schema. This can take several forms:

• Linearized Triples: For relation or event extraction, output is sequenced as: $$[\mathrm{<sub>}, \mathrm{subject}, \mathrm{<rel>}, \mathrm{relation}, \mathrm{<obj>}, \mathrm{object}, \mathrm{<et>}]$$ for each triple (Josifoski et al., 2021, Whitehouse et al., 2023).
• JSON Serialization: For entity and event extraction, outputs are formatted as JSON objects containing span text, offsets, and attributes, enabling natural mapping to downstream knowledge graph or slot-filling tasks (Townsend et al., 2021, Choi et al., 20 Apr 2025, Ying et al., 30 Jan 2025).
• Prompt and Slot-Filling: Templates with type-specific or modular sub-prompts are constructed, with slot placeholders to be filled auto-regressively by the LLM (Kan et al., 2022). This formulation enhances generalization, especially in low-resource and compositional settings.
• Two-Stage Generation and Schema Injection: Some designs explicitly separate the generation of structural elements (e.g., term–status pairs in dialogues) and inject schema knowledge or constraints via prompts, facilitating fine-grained control over output (Hu et al., 2023).

5. Evaluation Methods and Empirical Results

GIE methods are evaluated on diverse benchmarks across newswire (CoNLL03, ACE04/05), biomedical (GENIA, BC5CDR), social media (WNUT17), scientific text (SciERC), multimodal documents (SROIE, FUNSD, CORD), and domain-specific datasets (finance, medical).

Metrics include:

• Micro-averaged Precision, Recall, F1 over token, span, or triple-level structures. For generative code or structured outputs, BLEU and ROUGE-L are also employed.
• For constrained triple extraction, micro/macro F1 is also computed across all predicted and gold triplets (Xu et al., 2023).

Empirical evidence establishes:

• Supervised fine-tuning of seq2seq LLMs typically yields F1 performance in the low 90% range for NER on benchmark corpora (UIE (T5-large) ≈93.0 F1 on CoNLL03; InstructUIE (Flan-T5-11B) ≈92.9 F1).
• Few-shot/zero-shot in-context learning (e.g., Code4UIE, GPT-NER) yield F1 scores in the 55–75% range, below fully supervised methods but superior in low-resource conditions.
• GIE models such as GenIE and WebIE achieve high accuracy and scalability in closed triple extraction (GenIE: 88.2% micro F1 for Wiki-NRE, 68.9% on large REBEL schema) and demonstrate improved robustness and generalizability with auxiliary entity-linking objectives or constrained decoding (Josifoski et al., 2021, Whitehouse et al., 2023).
• In multimodal KIE, generative models like GenKIE and GMN outperform or match discriminative labelers, with superior robustness to OCR noise and no reliance on expensive token-level annotation (Cao et al., 2023, Cao et al., 2022).
• GIE maintains superior performance in extracting overlapping, nested, or compositional entities, and enables more faithful and interpretable outputs in financial and biomedical settings (Choi et al., 20 Apr 2025, Ying et al., 30 Jan 2025, Hsu et al., 2024).

6. Technical Challenges and Ongoing Research

Key challenges and open questions in GIE include:

• Universal Schema Alignment: Unified models must handle long contexts, variable structure, and cross-task misalignment. Current models degrade with increasing context length; integrating task-specific priors (e.g., argument–role correlations) may mitigate this (Xu et al., 2023).
• Prompt Design and Optimization: Although prompting is central to GIE, systematic principles for constructing, optimizing, and verifying effective prompts are not yet well established. Prompt selection remains heuristic, with variable transferability across domains and architectures (Xu et al., 2023).
• Low-Resource Transfer: In few-shot or domain-adaptation settings, effective retrieval and ordering of in-context examples is vital. Pipeline approaches (data augmentation, semi-supervised labeling) and composable prompts improve outcomes, but the field lacks optimal strategies (Xu et al., 2023, Kan et al., 2022).
• Faithful and Robust Triple Extraction: Generative models may hallucinate unsupported facts; bi-level constrained decoding, contrastive calibration, and auxiliary entity-linking objectives have been proposed to enhance faithfulness and reduce spurious generations (Josifoski et al., 2021, Whitehouse et al., 2023, Ye et al., 2020).
• Scalability and Efficiency: Models must balance long-context handling (e.g., document-length input), inference cost, and resource constraints, particularly in clinical and financial domains (Townsend et al., 2021, Ying et al., 30 Jan 2025).
• Output Serialization and Canonicalization: The design of output schema (e.g., fixpoint linearizations, permutation invariance) impacts downstream usability. Systematic studies of output format stability and end-to-end canonicalization are ongoing (Townsend et al., 2021, Kan et al., 2022).
• Extension to New Modalities and Structures: Recent models address visual layouts, table extraction, and code-based schemas, but extending GIE to cross-modal and non-textual settings remains a key research direction (Cao et al., 2023, Yang et al., 21 Mar 2025, Cao et al., 2022).

7. Impact, Implementations, and Resources

GIE has led to broad transformations in information extraction research and deployment:

• Open-Source Systems and Libraries: Practical GIE toolkits, such as LLM-IE, provide modular, schema-driven, and agent-assisted pipelines enabling biomedical and general-purpose IE with minimal custom engineering (Hsu et al., 2024).
• Universal, Prompt-Driven Models: Unification of NER, RE, EE, and more in a single system (e.g., LasUIE, InstructUIE, GENIE, Doc2Dict) enables efficient task scaling and rapid transfer to new domains (Fei et al., 2023, Ying et al., 30 Jan 2025, Townsend et al., 2021).
• Effective for Complex, Noisy Inputs: GIE shows superior handling of overlapping/ambiguous mentions, noisy OCR, visually complex formats, and emerging event/entity types (Choi et al., 20 Apr 2025, Cao et al., 2023, Cao et al., 2022).
• Empirical Milestones: Near or state-of-the-art scores on all major IE benchmarks, with fine-tuned generative LLMs establishing new SOTA on financial, medical, and document-level tasks (Choi et al., 20 Apr 2025, Ying et al., 30 Jan 2025).

A plausible implication is that GIE, driven by rapidly advancing LLM architectures and scalable prompt optimization, is positioned to supplant traditional pipeline-based IE approaches for heterogeneous, high-value extraction tasks in both research and industry. Future research will likely focus on robust schema generalization, resource-efficient adaptation, and integrating cross-modal knowledge representations (Xu et al., 2023).