KRCA: An Efficient Root Cause Analysis System in Hyper-Scale Microservice Systems via Agentic AI

Abstract: Hyper-scale microservice systems have become the standard infrastructure for large-scale Internet companies. These systems consist of numerous loosely coupled microservices that evolve independently through continuous development and deployment. Such complexity makes failures unavoidable, necessitating efficient Root Cause Analysis (RCA) to help Site Reliability Engineers (SREs) quickly localize root cause services and classify failure types. However, existing RCA methods often struggle to adapt to the extreme dynamism and massive scale of these systems. In this paper, we present KRCA, an end-to-end RCA system designed for hyper-scale microservice systems. To manage the vast search space, KRCA employs a multi-stage pipeline that begins with an API-level drilldown to isolate suspicious services. It then instantiates a skeleton-based causal graph from anomalous metrics to serve as a high-recall structural prior, before utilizing a memory-augmented multi-agent framework to verify causality and generate the final failure report. By combining structured causal constraints with multi-agent reasoning, KRCA employs balances diagnostic accuracy with the efficiency requirements of real-time production use. Experimental results show that KRCA achieves AC@1 scores of 0.88 and 0.79 for root cause service localization and failure type classification, outperforming the strongest baseline by at lease 31% in absolute gains. KRCA has been deployed in Kuaishou's production environment for over six months, reducing the average diagnosis time by 77.3%.

• The paper presents an agentic RCA framework that uses API-level drilldown and multi-agent LLM collaboration to achieve superior diagnostic accuracy.
• It employs a skeleton-based causal graph to effectively narrow the candidate search space and improve efficiency in handling high-dimensional observability data.
• Empirical results demonstrate significant performance gains, including a 77% reduction in incident diagnosis time and improved root cause localization.

KRCA: Agentic Root Cause Analysis for Hyper-Scale Microservice Systems

Introduction and Motivation

The paper "KRCA: An Efficient Root Cause Analysis System in Hyper-Scale Microservice Systems via Agentic AI" (2607.01788) addresses the core challenges of root cause analysis (RCA) within hyper-scale microservice deployments, characterized by both extreme dynamism and massive system scale. Contemporary internet-facing platforms increasingly rely on loosely coupled microservices with decentralized development and deployment, often involving hundreds of thousands of services and millions of observed metrics. The resultant system is highly dynamic, presenting frequent topological and behavioral changes, while the volume and velocity of observability data rapidly become prohibitive for manual or monolithic approaches to RCA.

Existing RCA solutions, whether based on deep learning or LLM-driven agents, are fundamentally constrained by the speed at which they can adapt to ongoing system evolution, the operational cost of maintenance and retraining, and the inherent limitations of LLMs to reason effectively over long, complex contexts. The compositional propagation of failures and the intricate metric interdependencies in such environments expose statistical causal discovery methods, direct LLM query pipelines, and agentic approaches to severe efficiency, accuracy, and hallucination bottlenecks.

Figure 1: Extreme dynamism and massive scale are dominant characteristics of hyper-scale microservice platforms, driving continuous system evolution and monitoring at an unprecedented scale.

Empirical Analysis of RCA Limitations

The authors provide concrete evidence from real-world production systems highlighting the limitations of current RCA methods. Service-level observability is shown to be too coarse for precise root cause localization, as API-level granularity is necessary to avoid the dilution and masking of critical anomalies due to traffic distribution over multiple APIs within a single service. Empirical results further demonstrate that as the number of anomalous metrics rises, the accuracy of classic causal discovery algorithms like PC and Granger causality drops precipitously, with performance degrading to below 20% accuracy in high-dimensional scenarios. Similarly, LLMs, when tasked directly with unconstrained metric sets, exhibit declining performance and increased hallucinations, particularly as context length grows.

Figure 2: Service-level (vs. API-level) dependency graphs significantly increase the downstream search space; classical and LLM-based causal discovery accuracy declines rapidly with the number of anomalous metrics.

The key insight is that effective RCA in hyper-scale environments requires narrowing the space of candidate services from the outset, leveraging API-level signals, and imposing structural priors to constrain LLM reasoning.

System Architecture

KRCA introduces a staged, modular architecture that processes failures with high efficiency and diagnostic fidelity. The system consists of three main modules:

1. API-Level Drilldown: Initiated from the alerting API, the system recursively scores and traverses dependency edges based on failure rate and latency signals, using a robust propagation score. This efficiently prunes the space of candidate root causes, focusing subsequent analysis only on the top-N suspicious services.
2. Skeleton-Based Causal Graph Instantiation: For each suspicious service, detected anomalous metrics are mapped into a generic, high-recall causal skeleton structured around meta-metric types (External, Internal, Dependency, KPI). This prior encodes validated propagation routes and semantic constraints, serving as a robust structural foundation for finer-grained diagnosis.
3. Multi-Agent Collaboration: A master-slave LLM-based architecture is used, where the Main Agent coordinates a suite of domain-expert Sub Agents (e.g., for Traffic, CPU, GPU, SQL). These agents iteratively verify, correct, and refine the initially generated causal graphs using evidence from the system's observability corpus. A memory-augmented retrieval module (RAG) supplies relevant historical context from a three-tier memory system—Working, Factual, and Experiential—to enable evidence-based, rapid, and context-aware reasoning.

Figure 3: The diagnostic pipeline comprises API-level drilldown, modular causal graph generation, and coordinated multi-agent refinement.

Figure 4: Main Agent and domain Sub Agents employ RAG-supported memory to iteratively verify and correct the causal graph, enabling parallelized reasoning specialized by metric domain.

Technical Implementation

The API-level drilldown mechanism combines failure rate and latency anomaly detection by exploiting synchronous propagation patterns in metric time series and measuring statistical correlation and anomaly degree, respectively. The scoring function ensures recursive traversal only pursues those downstream APIs most likely causally implicated. This sharply restricts candidate expansion, directly impacting both recall and efficiency.

Upon isolating suspicious services, the skeleton-based causal graph is generated. Metric anomalies are detected using a hybrid of Isolation Forest and PatternMatcher, addressing the unique noise patterns and diverse anomaly characteristics across metrics and services. The mapping of metrics to the meta-metric skeleton leverages prior domain knowledge of feasible failure propagation, dramatically reducing spurious causal edge inclusion and supporting downstream explainability.

The multi-agent collaboration stage decomposes the complex verification task into parallelized, domain-specialized roles, each with tailored diagnostic tools and prompts. The RAG mechanism leverages a compound similarity score balancing service, metric lexical similarity, and recency, supporting accurate retrieval from Factual and Experiential memory. This design is demonstrated to greatly increase the robustness and contextual accuracy of LLM reasoning while avoiding context-window overflow and minimizing hallucination.

Evaluation

The paper extensively evaluates KRCA using a dataset of 300 diverse, real-world failures from Kuaishou’s production system and in continuous production deployment across several months. Metrics include AC@1 and AC@3 for both root cause localization and failure type classification, as well as full end-to-end latency and human-rated report quality.

Key results:

• KRCA achieves AC@1 of 0.88 (root cause) and 0.79 (failure type), exceeding the best baseline (RCA-Agent) by absolute margins of 31% and 32%, respectively.
• Latency is substantially lower than unconstrained agentic baselines (e.g., 146s/case for KRCA vs. 342s/case for RCA-Agent).
• In production, average diagnosis time for incidents was reduced from 52 minutes to 11.8 minutes—a 77.3% decrease.
• SRE usability assessment yields mean readability and usefulness scores above 4.1/5, with high proportions (HIP>79%) rated as high utility.

Figure 5: Sensitivity analysis reveals that diagnostic accuracy is robust against variation in drilldown threshold, Top-N candidate count, and time lag parameter L.

The ablation study confirms that all components—API-level drilldown, skeleton-based structure, multi-agent reasoning, tiered memory, and structured RAG—make critical contributions to overall system efficacy and efficiency.

Deployment and Practical Impact

KRCA has been continuously deployed to Kuaishou's monitoring infrastructure (Tianwen) since October 2025, supporting over 2,000 alerts per day, with a correct root cause and failure type identification rate of 82% for emergency incidents. The authors present practical deployment lessons, emphasizing the importance of context engineering, prompt and tool calibration, error-handling, and progressive diagnostic result disclosure for maximizing operational trust and adoption rates among reliability engineers.

Figure 6: KRCA production deployment architecture integrates with monitoring and incident response platforms, supporting scalable real-time RCA across Kuaishou’s hyper-scale system.

A detailed case study exemplifies the substantial business impact of prompt, accurate RCA—preventing major monetary losses by expediting diagnosis in a real cascading storage failure.

Related Work and Theoretical Implications

KRCA distinguishes itself from prior approaches in both causal discovery and LLM-based RCA by its hybridization of structural prior encoding, efficient context space reduction, and agentic, evidence-based reasoning. It avoids the scalability bottlenecks of end-to-end LLM pipelines, statistical methods susceptible to high-dimensional noise, and monolithic ML approaches reliant on retraining. Its staged agentic design offers a transferable template for future RCA and observability automation research within even larger or more distributed system architectures.

Theoretically, the work substantiates the utility of imposing domain-constrained structural priors and modular reasoning to control the combinatorial complexity of AI-driven systems diagnosis, highlighting a path forward for LLM and agentic AI in production-critical, dynamic infrastructure contexts.

Conclusion

KRCA is a comprehensive agentic root cause analysis framework tailored for hyper-scale microservice systems. By integrating API-level drilldown, structured causal priors, and multi-agent LLM-based reasoning with advanced memory retrieval, KRCA achieves leading-edge diagnostic accuracy and operational efficiency, verified both in controlled experiments and at production scale. The system establishes a robust foundation for real-time, explainable incident diagnosis, delineating a direction for future research in scalable, agent-powered RCA and observability for complex distributed environments.