MIRAGE: Online LLM Simulation for Microservice Dependency Testing

Abstract: Existing approaches to microservice dependency simulation--record-replay, pattern-mining, and specification-driven stubs--generate static artifacts before test execution. We propose online LLM simulation, a runtime approach where the LLM directly answers each dependency request as it arrives, maintaining cross-request state throughout a test scenario. No mock specification is pre-generated; the model reads the dependency's source code, caller code, and production traces, then simulates dependency behavior on demand. We instantiate this approach in MIRAGE and evaluate it on 110 test scenarios spanning 14 caller-dependency pairs across three microservice systems (Google's Online Boutique, Weaveworks' Sock Shop, and a custom system). In white-box mode (dependency source available), MIRAGE achieves 99% status-code fidelity (109/110) and 99% response-shape fidelity, compared to 62% / 16% for record-replay. End-to-end, caller integration tests produce the same pass/fail outcomes with MIRAGE as with real dependencies (8/8 scenarios). A signal ablation shows dependency source code is often sufficient for high-fidelity runtime simulation (100% alone); without it, the model still infers correct error codes (94%) but loses response-structure accuracy (75%). Constraining LLM output through typed intermediate representations reduces fidelity on complex stateful services (55%) while performing adequately on simple APIs (86%), suggesting that the runtime approach's implicit state tracking matters for behavioral complexity. Results are stable across three LLM families (within 3%) at $0.16 to $0.82 per dependency.

• The paper introduces MIRAGE, an online LLM-driven approach that dynamically simulates microservice dependencies to overcome the limitations of static mocks.
• It demonstrates that MIRAGE achieves 99% status-code and response-shape fidelity in white-box mode, significantly outperforming record-replay and IR-based methods.
• The approach maintains stateful, real-time interactions across various microservice architectures, offering practical benefits for integration testing and CI pipelines.

Online LLM Simulation for Microservice Dependency Testing: An Analysis of MIRAGE

Introduction

Traditional microservice dependency simulation strategies rely on pre-generated artifacts—such as record-replay logs, trace-driven pattern extraction, or contract-based stubs—that attempt to encode downstream service behavior ahead of test execution. These static mock artifacts fail in the presence of scenario generalization, unseen parameters, or implicit and cross-request state, thus limiting test coverage and reliability. The paper "MIRAGE: Online LLM Simulation for Microservice Dependency Testing" (2604.04806) introduces online LLM-based simulation as a runtime testing primitive that addresses these limitations by leveraging instruction-following LLMs to synthesize dependency responses in real-time, maintaining an evolving interaction state across each test scenario.

MIRAGE System Architecture and Approach

MIRAGE is instantiatied as an online LLM-in-the-loop approach. For each incoming HTTP request, the LLM is presented with (a) the dependency’s implementation (white-box mode) or caller code and traces (black-box/ablation modes), augmented with test scenario context and accumulated conversation history. The model produces HTTP responses—including correct status codes and structured JSON bodies—in a turn-by-turn simulation, updating the context history to maintain implicit state.

MIRAGE consists of three main components: a Context Builder that generates LLM prompts using code and trace signals, a per-request LLM Server that manages bidirectional request/response messaging and stateful history, and a Scenario Context mechanism for injecting sequence information relevant to the ongoing test (enabling anticipation of multi-step flows without target responses). For evaluation, 110 test scenarios across three microservice systems (Google Online Boutique, Weaveworks Sock Shop, and a custom Demo system) are used, covering non-trivial behavioral categories such as stateful lifecycles, code-path generalization, and error handling.

Experimental Results

The main empirical results demonstrate a strong advantage for MIRAGE’s runtime simulation, especially in white-box mode, where the LLM is provided with dependency source code.

Figure 1: MIRAGE attains 99% status-code and 99% response-shape fidelity across all three benchmarks, substantially outperforming record-replay and pattern/IR-based baselines.

• Status-Code Fidelity: MIRAGE (white-box) achieves 99% fidelity ($109/110$), outperforming record-replay (62%) and all IR-based approaches.
• Response-Shape Fidelity: MIRAGE (white-box) also matches 99% of real service top-level response structures, compared to only 16% from record-replay methods.
• Black-Box and Ablation Modes: In the absence of dependency source, MIRAGE still maintains 94% status-code fidelity, with body-shape fidelity dropping to 75%, highlighting a critical failure mode for body-aware consumers of mocks.

Notably, for end-to-end integration tests, caller processes exhibit identical pass/fail outcomes on MIRAGE mocks and real dependencies (8/8 scenarios), confirming the practical adequacy of the approach for system-level testing. The per-category performance breakdown shows MIRAGE achieves 100% accuracy on all stateless and most stateful, code-reasoning, and error-handling categories.

Structured IR Generation vs. Online Simulation

A critical evaluation contrasts MIRAGE’s free-form generation with explicit typed IR approaches, where the LLM is used only to generate a finite-state simulator prior to test execution.

Figure 2: While IR constraints achieve moderate fidelity on error handling, they collapse on scenarios requiring implicit or latent state modeling compared to MIRAGE’s 100% fidelity in stateful categories.

IR-constrained generation achieves only 55% overall fidelity (and 29% on stateful flows) on the Demo benchmark, while MIRAGE maintains 100% in the same categories. For simple APIs (e.g., the OB product-catalog), IR approaches achieve up to 86%. The main source of IR failure is inability to capture implicit or cross-request temporal state, reinforcing the value of online, conversational context for runtime interpreters.

Signal Ablation and Input Signal Analysis

An extensive signal ablation study quantifies the contribution of different sources of information (dependency source, caller code, traces).

Figure 3: Status-code fidelity remains robust above 92% across configurations, but body-shape fidelity is highly sensitive to the presence of dependency source, dropping to 53–69% when unavailable.

• Dependency Source Code Only: Achieves 100% fidelity for both metrics.
• Caller Code Only or Traces Only: Status fidelity remains above 92% but body-shape fidelity significantly degrades (53–69%).
• Black-Box (Caller + Traces): Status fidelity is still high (94%), but response body fidelity drops sharply, indicating that while LLMs can infer control flow from limited signals, precise output structure often depends on access to implementation details.

Specific failure analyses show that absence of dependency source is especially problematic for response structure (e.g., missing/extra JSON keys, incorrect nesting) and for scenarios where preset state initialization cannot be inferred from traces or caller logic.

Cross-Benchmark Consistency, Model Variance, and Economic Analysis

The MIRAGE approach delivers consistently high accuracy across three microservice systems, despite variation in language, state management, and behavioral complexity. Evaluation with multiple LLM families (Claude Opus/Sonnet, Kimi, MiniMax M2) confirms within-3% fidelity margin among frontier models, with Sonnet yielding a significant cost reduction (from \$0.82 to \$0.16 per dependency per suite), making it favorable for CI pipelines at scale.

LLM-in-the-loop latency per request is in the ~3s range, resulting in full-suite runtimes comparable to, or even favorable against, full infrastructure provisioning (Docker, DB setup) in continuous integration settings, especially given the removal of environment tear-down costs.

Practical Considerations and Limitations

MIRAGE’s body-shape fidelity must not be conflated with value-level correctness; the system verifies only top-level JSON key structure, not domain-specific scalar value agreement. Value-level checks on single-object endpoints show 100% agreement, while list-valued responses can diverge in element count, a recognized limitation for exhaustive data validation. The current prototype, using HTTP/JSON and scenario context injection, does not yet generalize to gRPC, message queues, or event-driven protocols.

The overall onboarding effort is moderate (~8 hours per system for traces, scenarios, and harness setup), with no manual prompt engineering required due to the automated context builder.

Theoretical and Practical Implications

The introduction of MIRAGE reconfigures the microservice testing landscape by decoupling mock fidelity from pre-generation completeness, instead emphasizing on-demand code and trace reasoning with LLM-powered stateful simulation. Practically, this increases the coverage and reliability of integration testing, especially in environments where downstream deployments are brittle, unavailable, or maintained by external teams. Theoretically, MIRAGE exposes the inherent limits of IR-based abstractions for complex systems and suggests the future utility of hybrid generative+verificational approaches in software testing and program synthesis.

The main contradictory claim to traditional software engineering dogma is that typed and structured contract extraction can actually hurt simulator fidelity in the presence of significant runtime state, inverting the conventional assumption that structure and constraint always improve safety and verifiability.

Future Research Directions

Potential avenues include:

• Extension of MIRAGE to support richer protocols (e.g., gRPC, evented architectures).
• Combination with post-hoc IR validators for value-level assurance.
• Integration with code search and retrieval-augmented generation for large or partially observed systems.
• Broader application beyond microservices, e.g., as LLM-powered OS kernel simulators or for hardware-in-the-loop testing.

Conclusion

The paper empirically demonstrates that online LLM-based dependency simulation, as realized in MIRAGE, substantially outperforms static, pre-generated mocks by providing high-fidelity, stateful, and body-aware service surrogates for microservice integration testing. The approach is cost-effective, compatible with mainstream LLMs, and robust across diverse microservice architectures. The results advocate for a paradigm shift toward runtime, context-sensitive simulation for service testing and validation, with clear guidance for deployment in both black- and white-box scenarios. The research highlights the need for future advances in hybrid simulation-verification, protocol coverage, and value-precise fidelity—key challenges at the intersection of AI and software engineering.