Papers
Topics
Authors
Recent
Search
2000 character limit reached

Knowledge Pyramid Construction for Multi-Level Retrieval-Augmented Generation

Published 31 Jul 2024 in cs.AI and cs.CL | (2407.21276v3)

Abstract: This paper addresses the need for improved precision in existing knowledge-enhanced question-answering frameworks, specifically Retrieval-Augmented Generation (RAG) methods that primarily focus on enhancing recall. We propose a multi-layer knowledge pyramid approach within the RAG framework to achieve a better balance between precision and recall. The knowledge pyramid consists of three layers: Ontologies, Knowledge Graphs (KGs), and chunk-based raw text. We employ cross-layer augmentation techniques for comprehensive knowledge coverage and dynamic updates of the Ontology schema and instances. To ensure compactness, we utilize cross-layer filtering methods for knowledge condensation in KGs. Our approach, named PolyRAG, follows a waterfall model for retrieval, starting from the top of the pyramid and progressing down until a confident answer is obtained. We introduce two benchmarks for domain-specific knowledge retrieval, one in the academic domain and the other in the financial domain. The effectiveness of the methods has been validated through comprehensive experiments by outperforming 19 SOTA methods. An encouraging observation is that the proposed method has augmented the GPT-4, providing 395% F1 gain by improving its performance from 0.1636 to 0.8109.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Knowledge graphs as context sources for llm-based explanations of learning recommendations. Preprint, arXiv:2403.03008.
  2. Lingua: Addressing scenarios for live interpretation and automatic dubbing. In Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track), pages 202–209.
  3. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136.
  4. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 78–106.
  5. Fine-tuning large enterprise language models via ontological reasoning. In International Joint Conference on Rules and Reasoning, pages 86–94. Springer.
  6. Iterative zero-shot llm prompting for knowledge graph construction. arXiv preprint arXiv:2307.01128.
  7. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45.
  8. Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning. Preprint, arXiv:2310.15205.
  9. Adapting large language models via reading comprehension. In The Eleventh International Conference on Learning Representations.
  10. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  11. Chatlaw: Open-source legal large language model with integrated external knowledge bases. Preprint, arXiv:2306.16092.
  12. A survey on in-context learning. Preprint, arXiv:2301.00234.
  13. Open information extraction from the web. Communications of the ACM, 51(12):68–74.
  14. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
  15. Openagi: When llm meets domain experts. Advances in Neural Information Processing Systems, 36.
  16. Re2g: Retrieve, rerank, generate.
  17. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints.
  18. Lora: Low-rank adaptation of large language models.
  19. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.
  20. Lawyer llama technical report. Preprint, arXiv:2305.15062.
  21. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. 1:2.
  22. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
  23. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
  24. Scatter: selective context attentional scene text recognizer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11962–11972.
  25. Knowledge-augmented language model and its application to unsupervised named-entity recognition. In North American Chapter of the Association for Computational Linguistics.
  26. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747.
  27. Query rewriting in retrieval-augmented large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  28. Text2kgbench: A benchmark for ontology-driven knowledge graph generation from text. In International Semantic Web Conference, pages 247–265. Springer.
  29. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
  30. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  31. Ontochatgpt information system: Ontology-driven structured prompts for chatgpt meta-learning. arXiv preprint arXiv:2307.05082.
  32. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering.
  33. Improving language understanding by generative pre-training.
  34. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. arXiv preprint arXiv:2310.10501.
  35. Malik Sallam. 2023. Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, volume 11, page 887. MDPI.
  36. Plaid: an efficient engine for late interaction retrieval.
  37. When flue meets flang: Benchmarks and large pretrained language model for financial domain. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
  38. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31.
  39. One embedder, any task: Instruction-finetuned text embeddings.
  40. Gemini: a family of highly capable multimodal models.
  41. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  42. Llama 2: Open foundation and fine-tuned chat models.
  43. From human experts to machines: An llm supported approach to ontology and knowledge graph construction.
  44. Fingpt: Instruction tuning benchmark for open-source large language models in financial datasets. NeurIPS Workshop on Instruction Tuning and Instruction Following.
  45. Minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th International Conference on Neural Information Processing Systems.
  46. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1–34.
  47. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
  48. Ross Wilkinson and Philip Hingston. 1991. Using the cosine measure in a neural network for document retrieval. In Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, pages 202–210.
  49. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564.
  50. A brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5):1122–1136.
  51. Retrieve-rewrite-answer: A kg-to-text enhanced llms framework for knowledge graph question answering. ArXiv, abs/2309.11206.
  52. Retrieve-rewrite-answer: A kg-to-text enhanced llms framework for knowledge graph question answering. arXiv preprint arXiv:2309.11206.
  53. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.07597.
  54. Pixiu: A large language model, instruction data and evaluation benchmark for finance. Preprint, arXiv:2306.05443.
  55. Mentallama: interpretable mental health analysis on social media with large language models. pages 4489–4500.
  56. Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue.
  57. Disc-lawllm: Fine-tuning large language models for intelligent legal services. Preprint, arXiv:2309.11325.
  58. Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309.10313.
  59. Alpacare: Instruction-tuned large language models for medical application. arXiv preprint arXiv:2310.14558.

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 3 tweets with 1 like about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free