Leveraging Large Language Models for Tacit Knowledge Discovery in Organizational Contexts

Abstract: Documenting tacit knowledge in organizations can be a challenging task due to incomplete initial information, difficulty in identifying knowledgeable individuals, the interplay of formal hierarchies and informal networks, and the need to ask the right questions. To address this, we propose an agent-based framework leveraging LLMs to iteratively reconstruct dataset descriptions through interactions with employees. Modeling knowledge dissemination as a Susceptible-Infectious (SI) process with waning infectivity, we conduct 864 simulations across various synthetic company structures and different dissemination parameters. Our results show that the agent achieves 94.9% full-knowledge recall, with self-critical feedback scores strongly correlating with external literature critic scores. We analyze how each simulation parameter affects the knowledge retrieval process for the agent. In particular, we find that our approach is able to recover information without needing to access directly the only domain specialist. These findings highlight the agent's ability to navigate organizational complexity and capture fragmented knowledge that would otherwise remain inaccessible.

• The paper demonstrates an LLM agent framework that iteratively documents tacit knowledge via employee interactions in simulated organizations.
• It uses an agent-based inquiry process modeled as a Markov Decision Process to optimize knowledge acquisition through SI model simulations.
• Experiments with 864 simulations and 94.9% full-knowledge recall highlight the influential role of informal connections in enhancing documentation efficacy.

Leveraging LLMs for Tacit Knowledge Discovery in Organizational Contexts

Introduction

The effective management and dissemination of tacit knowledge within organizations are formidable challenges due to the decentralized nature of such knowledge. The paper presents an agent-based framework employing LLMs to facilitate tacit knowledge documentation through iterative interactions with employees, thereby overcoming organizational knowledge dissemination hurdles. This research models the knowledge flow as a Susceptible-Infectious (SI) process using simulations across various synthetic company structures.

Methodology

Agent-Based Framework

The proposed framework delineates an agent-driven inquiry process within a dynamic organizational environment. The agent, based on LLMs, navigates through corporate hierarchies, gathering fragmented insights from employees to reconstruct dataset descriptions. The agent's operation follows an implicit Markov Decision Process (MDP), comprising knowledge states, actions, probabilistic transitions, and reward functions. This formulation aims to maximize accumulated rewards by optimizing knowledge acquisition strategies.

Figure 1: The agent iteratively builds knowledge and decides on its next course of action as it interacts with company employees in a conversation loop.

Simulation of Organizational Structures

Knowledge propagation within organizations is simulated through the SI model, representing individuals as nodes in formal and informal networks. Parameters such as hierarchical depth, number of employees, alpha (transmission probability), decay (knowledge waning rate), and informal connection count are varied to emulate diverse organizational environments. This simulation assesses the agent's ability to retrieve distributed knowledge effectively.

Implementation and Evaluation

Experiments encompassed 864 simulations with diverse organizational setups to measure the agent's efficacy. The evaluation focused on metrics like full-knowledge recall, METEOR scores, G-Eval coherence and faithfulness, and self-critical scores with context. With 94.9% full-knowledge recall, the approach demonstrates robust performance, showcasing its ability to capture and align fragmented knowledge using LLM-based agents.

Results

The paper's experimental analysis shows a strong positive correlation between informal connections within an organization and improved performance metrics. Additionally, the simulation results indicate that enhancements in the transmission rate and knowledge decay influence the agent's efficiency in achieving accurate table descriptions without reaching a central knowledge holder. Specifically, the approach achieved notable results in generating coherent and faithful textual descriptions of datasets within the organizational context.

Figure 2: Hierarchy and knowledge levels.

Discussion

The findings underline the potential of LLMs in automating knowledge extraction processes in organizational settings characterized by complex structures and communication pathways. Through effective dialogue-based interactions with simulated employees, the LLM-driven agent dynamically adjusts its knowledge states to enhance documentation quality. This research underscores the implications for deploying AI agents that bridge the gap between distributed knowledge and centralized documentation, enhancing overall organizational intelligence.

Conclusion

The study presents significant strides in leveraging LLMs for autogenous knowledge retrieval in organizational environments. The proposed framework's capability to reconstruct comprehensive data descriptions without extensive reliance on domain specialists points to promising avenues for integrating LLMs into corporate knowledge management systems. Future research could extend these insights into real-world applications, exploring the scalability of this approach in diverse industry contexts.