PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering
Abstract: Long-term memory plays a critical role in personal interaction, considering long-term memory can better leverage world knowledge, historical information, and preferences in dialogues. Our research introduces PerLTQA, an innovative QA dataset that combines semantic and episodic memories, including world knowledge, profiles, social relationships, events, and dialogues. This dataset is collected to investigate the use of personalized memories, focusing on social interactions and events in the QA task. PerLTQA features two types of memory and a comprehensive benchmark of 8,593 questions for 30 characters, facilitating the exploration and application of personalized memories in LLMs. Based on PerLTQA, we propose a novel framework for memory integration and generation, consisting of three main components: Memory Classification, Memory Retrieval, and Memory Synthesis. We evaluate this framework using five LLMs and three retrievers. Experimental results demonstrate that BERT-based classification models significantly outperform LLMs such as ChatGLM3 and ChatGPT in the memory classification task. Furthermore, our study highlights the importance of effective memory integration in the QA task.
- Richard C Atkinson and Richard M Shiffrin. 1968. Human memory: A proposed system and its control processes. In Psychology of learning and motivation, volume 2, pages 89–195. Elsevier.
- Qwen technical report.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278.
- Open question answering over tables and text. arXiv preprint arXiv:2010.10439.
- Hybridqa: A dataset of multi-hop question answering over tabular and textual data. arXiv preprint arXiv:2004.07347.
- Dialogsum challenge: Summarizing real-life scenario dialogues. In Proceedings of the 14th International Conference on Natural Language Generation, pages 308–313.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- Michael W Eysenck and Mark T Keane. 2020. Cognitive psychology: A student’s handbook. Psychology press.
- Topical-chat: Towards knowledge-grounded open-domain conversations. arXiv preprint arXiv:2308.11995.
- Chatdb: Augmenting llms with databases as their symbolic memory. arXiv preprint arXiv:2306.03901.
- Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.
- Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
- Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.
- Prompted LLMs as Chatbot Modules for Long Open-domain Conversation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4536–4554. ArXiv:2305.04533 [cs].
- Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
- Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
- Introduction to information retrieval. Cambridge university press.
- Hybridialogue: An information-seeking dialogue dataset grounded on tabular and textual data. arXiv preprint arXiv:2204.13243.
- Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266.
- Okapi at trec-3. Nist Special Publication Sp, 109:109.
- Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652.
- Llama 2: Open foundation and fine-tuned chat models.
- Endel Tulving and Fergus IM Craik. 2000. The Oxford handbook of memory. Oxford University Press.
- A survey of the evolution of language model-based dialogue systems. arXiv preprint arXiv:2311.16789.
- Beyond goldfish memory: Long-term open-domain conversation. arXiv preprint arXiv:2107.07567.
- Beyond Goldfish Memory: Long-Term Open-Domain Conversation. ArXiv:2107.07567 [cs].
- Ming Xu. 2023. Text2vec: Text to vector toolkit. https://github.com/shibing624/text2vec.
- Long time no see! open-domain conversation with long-term persona memory. arXiv preprint arXiv:2203.05797.
- Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
- Glm-dialog: Noise-tolerant pre-training for knowledge-grounded dialogue generation. arXiv preprint arXiv:2302.14401.
- Memory-augmented llm personalization with short-and long-term memory coordination. arXiv preprint arXiv:2309.11696.
- Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112.
- Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243.
- UniMC: A Unified Framework for Long-Term Memory Conversation via Relevance Representation Learning. ArXiv:2306.10543 [cs].
- Personalized dialogue generation with diversified traits. arXiv preprint arXiv:1901.09672.
- Memorybank: Enhancing large language models with long-term memory. arXiv preprint arXiv:2305.10250.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.