Toward Conversational Agents with Context and Time Sensitive Long-term Memory

Abstract: There has recently been growing interest in conversational agents with long-term memory which has led to the rapid development of LLMs that use retrieval-augmented generation (RAG). Until recently, most work on RAG has focused on information retrieval from large databases of texts, like Wikipedia, rather than information from long-form conversations. In this paper, we argue that effective retrieval from long-form conversational data faces two unique problems compared to static database retrieval: 1) time/event-based queries, which requires the model to retrieve information about previous conversations based on time or the order of a conversational event (e.g., the third conversation on Tuesday), and 2) ambiguous queries that require surrounding conversational context to understand. To better develop RAG-based agents that can deal with these challenges, we generate a new dataset of ambiguous and time-based questions that build upon a recent dataset of long-form, simulated conversations, and demonstrate that standard RAG based approaches handle such questions poorly. We then develop a novel retrieval model which combines chained-of-table search methods, standard vector-database retrieval, and a prompting method to disambiguate queries, and demonstrate that this approach substantially improves over current methods at solving these tasks. We believe that this new dataset and more advanced RAG agent can act as a key benchmark and stepping stone towards effective memory augmented conversational agents that can be used in a wide variety of AI applications.

• The paper introduces a hybrid retrieval model combining vector and tabular search to enhance meta-data queries in conversational memory.
• It presents a novel dataset simulating long-form dialogues with challenges like time-based and ambiguous queries.
• Results show up to 90% recall improvement, underscoring significant advancements for context-aware conversational AI.

"Toward Conversational Agents with Context and Time Sensitive Long-term Memory" (2406.00057)

Introduction

The paper addresses the development of conversational agents augmented with long-term memory, leveraging retrieval-augmented generation (RAG). The focus is on overcoming challenges inherent in retrieving information from long-form conversational data, contrasting with the retrieval tasks from static databases like Wikipedia. It identifies two primary challenges: handling time/event-based queries and managing ambiguous queries dependent on conversational context. A novel dataset is introduced to test these capabilities, accompanied by an advanced retrieval model integrating tabular search methods, vector-database retrieval, and prompting strategies.

Technical Contributions

Challenges in Conversational Contexts

The authors articulate two significant challenges facing RAG systems in conversational agents:

Conversational Meta-Data Based Queries: Unlike static database queries, conversational agents often encounter queries related to meta-data such as time, date, or speaker identity. The ability to retrieve information based on this meta-data is critical, as queries may not explicitly specify the content but depend on contextual metadata.

Ambiguous Questions: Ordinary generation tasks for LLMs handle pronoun and demonstrative ambiguity by relying on context. However, RAG systems face difficulties as ambiguous queries can mislead the retrieval mechanisms without understanding preceding context.

Figure 1: Examples of queries for various types in our dataset.

Dataset Generation

The study expands upon existing datasets of long-form dialogues, creating a test set that directly evaluates conversational agents' ability to manage the two identified types of queries. Modifications to the LoCoMo dataset include extending dialogues to prevent context fitting, padding sessions, and simulating realistic time-based responses for detailed temporal testing.

Advanced Retrieval Model

A key innovation is the retrieval model combining vector-based semantic queries with tabular query methods. This model uses a chain-of-tables (CoTable) approach, enhancing retrieval from chat logs based on meta-data requirements. The model uses classifiers to determine whether to employ semantic or meta-data retrieval methods, significantly improving the retrieval accuracy rate.

Figure 2: Depiction of our combined tabular and semantic vector-search method.

Results

The experimental results demonstrate substantial performance improvements in retrieving responses associated with specific meta-data. The combined CoTable and semantic approach provided recall improvements up to 90% across queries involving both meta-data and content. This exceeded the performance of traditional semantic retrieval methods, which struggled to handle meta-data-based queries.

Figure 3: F2 scores for each individual time-based test and the time+content based test. All models use k=10 for semantic search. Error bars show std. of recall and precision across data in each test.

Implications and Future Directions

Practical Implications

The development of conversational agents capable of effectively managing context-sensitive, meta-data-driven retrieval opens pathways for more sophisticated applications in personal assistants and customer service automation. These systems can provide consistent, context-aware interactions over extended periods, improving user experience and functional reliability.

Theoretical Implications

From a research perspective, the integration of CoTable methods with RAG architectures offers new insights into hybrid retrieval systems in AI. These findings could influence future research on conversational AI, particularly in optimizing retrieval strategies in dynamically changing conversational contexts.

Future Research

The results suggest several avenues for further inquiry, including refining classifiers for better semantic/meta-data differentiation, exploring alternative modeling techniques for ambiguous queries, and expanding dataset scenarios to include richer meta-data dimensions.

Conclusion

The paper "Toward Conversational Agents with Context and Time Sensitive Long-term Memory" presents compelling advances in retrieval techniques for conversational agents, addressing unique challenges posed by long-form dialogue data. With enhancements in dataset robustness and retrieval model architecture, the study sets a foundation for future advancements in AI systems equipped with context-sensitive long-term memory.