A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models

Abstract: Text summarization research has undergone several significant transformations with the advent of deep neural networks, pre-trained LLMs (PLMs), and recent LLMs. This survey thus provides a comprehensive review of the research progress and evolution in text summarization through the lens of these paradigm shifts. It is organized into two main parts: (1) a detailed overview of datasets, evaluation metrics, and summarization methods before the LLM era, encompassing traditional statistical methods, deep learning approaches, and PLM fine-tuning techniques, and (2) the first detailed examination of recent advancements in benchmarking, modeling, and evaluating summarization in the LLM era. By synthesizing existing literature and presenting a cohesive overview, this survey also discusses research trends, open challenges, and proposes promising research directions in summarization, aiming to guide researchers through the evolving landscape of summarization research.

• The paper systematically categorizes text summarization into four paradigms evolving from statistical methods to advanced LLMs.
• It details methodological advancements across extractive, abstractive, and hybrid approaches using metrics like ROUGE and BERTScore.
• It highlights challenges such as hallucinations and factuality, proposing multimodal and interactive summarization as future directions.

Systematic Survey of Text Summarization: From Statistical Methods to LLMs

The paper "A Systematic Survey of Text Summarization: From Statistical Methods to LLMs" provides an extensive review of the evolution in text summarization techniques. This survey charts the progression from traditional statistical methods to the current use of LLMs, divided into four key paradigms.

Major Paradigms in Summarization

The paper categorizes the evolution of text summarization into four distinct paradigms: statistical methods, deep learning-based methods, pre-trained LLM (PLM) fine-tuning, and recent advances in LLMs.

Figure 1: The Evolution of the Four Major Paradigms in Text Summarization Research.

1. Statistical Methods: Initial efforts focused largely on extractive summarization using heuristic, optimization, and graph-based methods. Techniques such as TextRank and LexRank developed graph-based models to rank sentences, demonstrating the utility of graph centrality for extractive summarization.
2. Deep Learning Era: This era marked a shift towards utilizing neural networks, especially recurrent and convolutional neural networks, to enhance contextual understanding and sequence modeling. Reinforcement learning techniques further improved extractive and abstractive summarization by addressing exposure bias.
3. PLM Fine-Tuning: Models like BERT and BART brought significant performance improvements through the "pre-train, then fine-tune" paradigm, using vast textual data for language representation.
4. LLMs: The current paradigm explores the capabilities of models like GPT and their advanced generative capabilities. LLMs offer robust performance in zero-shot and few-shot scenarios, although challenges such as hallucinations persist.

Approaches and Architectures

Figure 2: Categorization of Summarization Approaches based on input formats and output styles.

Text summarization methods are categorized based on input (single-document, multi-document, and query-focused) and output (extractive, abstractive, and hybrid) styles. These categories define the structure and focus of summarization tasks.

Extractive vs Abstractive

• Extractive Summarization: This approach involves selecting important sentences or phrases directly from the source text. Classical and NN-based methods often frame this as a sequence labeling problem.
• Abstractive Summarization: In contrast, abstractive methods generate new sentences that can introduce novel words. This approach leverages models like BART and T5, which excel in seq2seq tasks using encoder-decoder architectures, as seen in Figure 3.

  Figure 3: Illustration of the CNN model architecture (left) and the RNN model architecture (right) (taken from~\cite{li2022survey}).

Evaluation Metrics

The paper underscores the importance of using both human and automatic evaluation metrics such as ROUGE, BERTScore, and more nuanced QA-based metrics to assess summarization quality. The limitations of ROUGE in capturing semantic similarity are addressed, leading to the exploration of model-driven evaluation methods like BartScore.

Challenges and Future Directions

Despite the progress, challenges such as ensuring the faithfulness and factuality of summaries remain. Furthermore, hallucination in LLM outputs is a significant concern. Addressing these issues involves exploring more rigorous data annotation and better training paradigms, which are critical for practical real-world applications.

The survey also speculates on the transformative potential of LLMs in automating complex summarization tasks across diverse domains. The integration of multimodal and interactive summarization systems could redefine information consumption and decision-making processes.

Conclusion

This survey comprehensively captures the trajectory of text summarization research, highlighting how methodological advances have progressively elevated system performance. It serves as a pivotal reference for ongoing and future work in leveraging cutting-edge LLMs to tackle the multifaceted challenges of summarizing vast, diverse content efficiently and accurately. The field remains vibrant, with the potential for significant breakthroughs as computational models become more sophisticated and aligned with human cognitive processes.