CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation

Abstract: Retrieval-Augmented Generation (RAG) has become a powerful paradigm for enhancing LLMs through external knowledge retrieval. Despite its widespread attention, existing academic research predominantly focuses on single-turn RAG, leaving a significant gap in addressing the complexities of multi-turn conversations found in real-world applications. To bridge this gap, we introduce CORAL, a large-scale benchmark designed to assess RAG systems in realistic multi-turn conversational settings. CORAL includes diverse information-seeking conversations automatically derived from Wikipedia and tackles key challenges such as open-domain coverage, knowledge intensity, free-form responses, and topic shifts. It supports three core tasks of conversational RAG: passage retrieval, response generation, and citation labeling. We propose a unified framework to standardize various conversational RAG methods and conduct a comprehensive evaluation of these methods on CORAL, demonstrating substantial opportunities for improving existing approaches.

• The paper introduces CORAL, a benchmark that evaluates multi-turn conversational RAG systems using novel sampling strategies derived from Wikipedia.
• It assesses key aspects such as passage retrieval, response generation, and citation labeling, demonstrating performance gaps and scaling impacts.
• Experimental results reveal that current RAG systems struggle with complex dialogue dynamics, highlighting the need for improved conversational compression techniques.

CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation

Introduction

The paper "CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation" (2410.23090) introduces CORAL, a benchmark designed to evaluate Retrieval-Augmented Generation (RAG) systems in multi-turn conversational contexts. RAG is a prevailing paradigm that enhances LLMs like GPT-4 by integrating external knowledge retrieval to improve response accuracy. While RAG has been extensively investigated for single-turn applications, there remains a gap in addressing its effectiveness in multi-turn scenarios, increasingly seen in real-world systems. CORAL fills this gap, offering a comprehensive evaluation framework that focuses on passage retrieval, response generation, and citation labeling across complex conversational tasks sourced from Wikipedia.

Dataset Construction

CORAL's construction involves automatically deriving information-seeking conversations from Wikipedia, exploiting its structured content for realistic benchmarking of conversational RAG systems. The dataset captures diverse conversational flows using sampling strategies to mimic real dialogue dynamics.

Figure 1: Illustration of the four sampling strategies. The red arrows show the sampled conversation flow, with numerical labels on the nodes indicating the round of the sampled conversation turns.

Three strategies guide the conversation initialization: Linear Descent Sampling (LDS) for straightforward topic exploration, Sibling-Inclusive Descent Sampling (SIDS) for parallel topic interrogation, and Dual-Tree Random Walk (DTRW) for handling topic shifts.

Figure 2: Part (a) is an overview of the CORAL dataset construction process. The red arrows show the sampled conversation flow, with numerical labels on the nodes indicating the round of the sampled conversation turns. The content under each sampled (sub)title serves as the conversational response in CORAL. Part (b) is the three conversation compression strategies in conversational RAG.

Evaluation Tasks

CORAL evaluates conversational RAG systems across open-domain settings, emphasizing knowledge-intensity, free-form responses, and citation accuracy. Each task caters to key aspects necessary for real-world multi-turn dialogue systems: passage retrieval, response generation, and citation labeling.

• Passage Retrieval: Assesses the system's capability to extract relevant information amid conversational context changes.
• Response Generation: Tests the system's ability to generate accurate, context-rich answers.
• Citation Labeling: Ensures response transparency by requiring proper attribution of information sources.

Experimental Results

Experiments on CORAL demonstrate substantial performance gaps in current conversational RAG systems, underscoring opportunities for improving retrieval and generative capabilities. The benchmark allows for scaling analysis, revealing how citation accuracy benefits from larger model dimensions, but response generation hits a plateau past certain parameter thresholds.

Figure 3: The scaling analysis of generation and citation labeling performance.

Additionally, the evaluation of varied conversation history lengths highlights the challenges posed by redundant information and topic shifts. Fine-tuning models with conversation compression strategies improves both response generation and citation labeling.

Figure 4: Generation results of different conversation history length. The curve in the figure represents the ROUGE-L score. The histogram shows the results of GPT-4 scores comparing model-generated responses with golden responses.

Implications and Future Directions

CORAL provides critical insights into optimizing conversational RAG systems, aligning academic advancements with practical applications. By addressing the intricacies of multi-turn interactions, CORAL facilitates innovation that can bridge existing gaps and refine the precision of conversational AI. Future developments may explore enhanced conversation compression techniques and expand domain-specific conversational datasets to further improve RAG systems' adaptability and efficiency in diverse applications.

Figure 5: The GPT-4 evaluation score.

Conclusion

The introduction of CORAL marks a significant step forward in the systematic evaluation of conversational RAG systems under realistic, multi-turn conditions. It serves as a robust platform for identifying strengths and weaknesses in current methodologies, offering a path to advancements that could fundamentally improve interaction quality in AI-driven conversations. The dataset and its empirical evaluations open avenues for enhanced dialogue systems capable of navigating complex conversational landscapes with greater fidelity and accuracy.