The Extractive-Abstractive Spectrum: Uncovering Verifiability Trade-offs in LLM Generations

Abstract: Across all fields of academic study, experts cite their sources when sharing information. While LLMs excel at synthesizing information, they do not provide reliable citation to sources, making it difficult to trace and verify the origins of the information they present. In contrast, search engines make sources readily accessible to users and place the burden of synthesizing information on the user. Through a survey, we find that users prefer search engines over LLMs for high-stakes queries, where concerns regarding information provenance outweigh the perceived utility of LLM responses. To examine the interplay between verifiability and utility of information-sharing tools, we introduce the extractive-abstractive spectrum, in which search engines and LLMs are extreme endpoints encapsulating multiple unexplored intermediate operating points. Search engines are extractive because they respond to queries with snippets of sources with links (citations) to the original webpages. LLMs are abstractive because they address queries with answers that synthesize and logically transform relevant information from training and in-context sources without reliable citation. We define five operating points that span the extractive-abstractive spectrum and conduct human evaluations on seven systems across four diverse query distributions that reflect real-world QA settings: web search, language simplification, multi-step reasoning, and medical advice. As outputs become more abstractive, we find that perceived utility improves by as much as 200%, while the proportion of properly cited sentences decreases by as much as 50% and users take up to 3 times as long to verify cited information. Our findings recommend distinct operating points for domain-specific LLM systems and our failure analysis informs approaches to high-utility LLM systems that empower users to verify information.

• The paper introduces the extractive-abstractive spectrum to evaluate the balance between output utility and source verifiability.
• The paper quantifies that as outputs become more abstractive, perceived utility can rise by 200% while citation accuracy may drop over 50%.
• The paper recommends dynamic query routing to tailor LLM output styles for varying task stakes, optimizing both efficiency and reliability.

Navigating the Extractive-Abstractive Spectrum in LLM Generations

The paper "The Extractive-Abstractive Spectrum: Uncovering Verifiability Trade-offs in LLM Generations" presented by Theodora Worledge, Tatsunori Hashimoto, and Carlos Guestrin explores the critical balance between utility and verifiability within LLMs. This work introduces the "extractive-abstractive spectrum," a conceptual framework that identifies various operational stages between purely extractive systems (e.g., search engines) and fully abstractive models (e.g., current LLMs). Crucially, this study examines the implications of these operational points on information reliability, highlighting the contingent relationship between the informativeness of a model and its citation validity.

Overview

Central to the research is a survey that found a prevalent user preference for search engines over LLMs in high-stakes information retrieval scenarios, due to the entrenched need for source verifiability. This foundational finding laid the groundwork for scrutinizing intermediate operations across the extractive-abstractive spectrum. The research delineates five distinct operational points: extractive, quoted, paraphrased, entailed, and abstractive, each bearing distinct trade-offs between perceived utility and ease of verification.

Through meticulous human evaluations, the research compares these operational nations across seven systems over four diverse query distributions: web search, language simplification, multi-step reasoning, and medical advice. These comparisons confirmed significant quantitative trade-offs: as outputs become more abstractive, perceived utility may improve by 200%, while the citation coverage can decline by over 50%, and verification time required by users could triple.

Key Findings and Theoretical Implications

A noteworthy contribution of the paper is its demonstration of how citation verifiability decreases sharply as LLM outputs transition from extractive to abstractive modes. The research indicates substantial citation precision drops in abstractive systems due to the absence of pre-designated or inherently accompanying citations, as opposed to more extractive systems where verifiability is more robust. By advocating for a refined focus on citation identification practices, including post-hoc citations, the study promotes a strategic segmentation of query types that are best served by differing levels of abstraction. Importantly, it acknowledges that the task-specific needs may vary: high-stakes queries necessitate more verifiable outputs, whereas creative or open-ended queries may benefit from abstractive richness.

Practical Implications and Future Prospects

The paper has profound implications on the deployment and development of domain-specific LLM systems. Its recommendations emphasize the design of systems that judiciously alternate between operating points based on user requirements, task complexity, and domain specificity. By doing so, such systems can optimally leverage utility while maintaining necessary reliability. The research also proposes strategic routing of queries to distinct operational points to enhance user experience depending on the information needs, thus allowing systems to tailor their approach to provide not only information but trustworthiness.

In contemplating future work, the authors underscore the necessity for improvements in downstream understanding of citation identification and the establishment of systems that seamlessly integrate multiple operational points for diverse queries. This suggests an avenue for LLMs to evolve beyond a binary between abstraction and extraction, moving towards a fusion that enhances user trust without sacrificing the breadth of information — a potential future pathway that integrates efficiency, customizability, and verifiability.

Conclusion

This research is instrumental in driving forward our understanding of LLM capabilities, balancing utility and verifiability. The insights presented call for a reevaluation of our existing models and propose a nuanced approach to model deployment. By mapping the extractive-abstractive spectrum and elucidating the trade-offs involved, it offers a significant contribution to the field, presenting a roadmap for creating both efficient and reliable information retrieval systems.