When Local Monitors Miss Compositional Harm: Diagnosing Distributed Backdoors in Multi-Agent Systems

Abstract: As multi-agent, tool-using LLM systems are deployed, a common safety net is a runtime monitor that checks each message, tool call, or step on its own. We show this net has a fundamental hole. A distributed backdoor splits a harmful payload across agents, so every local check passes while the assembled object is the attack. The monitor can be right on every step and still miss the attack. The problem is not splitting itself: split fragments can still leak suspicious tokens or provenance edges. The hard case is \emph{local benignness}. No fragment carries the harm, and what is left looks like ordinary benign traffic. We formalize this as an \emph{observability boundary}: a monitor catches only what its view can tell apart from benign traffic. We prove that once the fragments look benign in the monitored view, no detector on that view can catch them, however strong it is. Across a controlled testbed, an external benchmark, and end-to-end agent runs, local monitors lose the signal exactly as local evidence disappears, and it returns only when the monitor sees the assembled object. A monitor trained only on benign traffic recovers the attack's code structure across held-out encodings (0.874 mean AUROC). A decoded-view gate, given the encoding family, blocks every tested attack. But seeing more is not enough: full-trace monitors and decoders still fail unless they reach the representation where the payload is exposed. Local safety is not global safety when harm is compositional, and the open problem is finding that representation.

• The paper establishes an observability boundary, showing that local monitors cannot detect attacks when malicious fragments appear statistically benign.
• It demonstrates through theory and experiments that local detectors achieve near-chance AUROC when evidence is fragmented and indistinguishable.
• It reveals that decoding the assembled payload is crucial for detection, shifting defense strategies from local classification to representation discovery.

Diagnosing Observability Boundaries for Distributed Backdoors in Multi-Agent Systems

Problem Definition and Formalization

This paper identifies a fundamental limitation in runtime safety monitoring for tool-using and multi-agent LLM systems. The central claim is that distributed backdoor attacks can evade local monitors by decomposing malicious payloads into benign fragments, which are individually indistinguishable from ordinary traffic. The authors formalize this as an observability boundary: a monitor can only detect harm that is statistically and semantically apparent in the portion of traffic it observes. When fragments are locally benign and locally indistinguishable in total variation from benign data—two key conditions defined precisely—no local detector, regardless of architecture or training regime, can reliably identify the attack.

The authors provide an exact theoretical bound: for any $\varepsilon$-locally indistinguishable attack, the detection advantage (difference between TPR and FPR) for any detector restricted to the local view is at most $\varepsilon$. That is, if the monitor's observable window does not contain detectable evidence, its performance is upper-bounded by the statistical similarity between benign and attack fragments, formalized with total variation distance.

Experimental Validation and Locality Sweeps

To empirically investigate when monitors fail, the authors construct a diagnostic testbed ("Collaborative Shadows") with a controllable axis: they tune the visibility of attack evidence within fragments from obvious clues (e.g., source-to-sink data paths, suspicious encodings) to complete benignness (fragments are indistinguishable from typical outputs). Monitoring strategies evaluated include taint/provenance analysis, statistical and neural per-step detectors, as well as assembly-based views (marker-controlled and marker-free).

Results are consistent across synthetic and external benchmarks (AgentDojo, DemonAgent):

• Local monitors only fail under local benignness and indistinguishability: When attack fragments carry visible cues, monitors succeed. The failure regime emerges only when fragments are both locally benign and statistically indistinguishable.
• AUROC values manifest locality limits: As local evidence disappears, per-step AUROC drops to chance ($0.5$), confirming the theoretical bound.
• Marker-free assembly views recover the attack: Monitors analyzing the assembled output (without knowledge of explicit attack markers) achieve high separability (mean AUROC $0.874$) even without attack training, demonstrating that payload structure becomes detectable post-composition.

Implications for Monitor Design and Attack Blocking

The paper systematically explores whether broader access, more fragments, or more sophisticated aggregation is itself sufficient to recover attacks missed by local monitors. The empirical findings are clear:

• Observation breadth is insufficient unless the representation is correct: Full-trace and codec-blind monitors, as well as common-decoder ensembles and LLM reconstruction probes, fail to block locally benign attacks unless they deduce and inspect the correct payload representation.
• Decoded-view gates block all attacks given coding knowledge: Runtime gates with knowledge of the encoding family (i.e., decoding the assembled fragments before execution) block $100\%$ ($0/50$ successful attack executions) in end-to-end tests across four open weights models, even at a $5\%$ benign FPR calibration.
• Monitors relying solely on code-likeness are defeated by benign coding activity: The marker-free assembly monitor can distinguish attack structure when the payload is code, but in the presence of benign code, only decoding the payload separates attacks from innocent activity (AUROC $1.000$ with the decoded-view gate, AUROC $0.090$ for codec-blind monitor).

Theoretical and Practical Frontier

The findings shift the locus of defensive effort from classifier improvement to representation discovery: the key challenge is not aggregating more fragments but ensuring the monitor observes the property where harm forms. The boundary holds even for advanced codec schemes (e.g., DemonAgent's dynamic encryption), demonstrating that the monitor's view, not detector type, is the bottleneck.

This recharacterizes benchmarking: "high detection rates" at the fragment level can mask monitor blindness to distributed, compositionally emergent harm. The authors urge a focus on monitoring strategies that expose and analyze the assembled representation, which may include code, plans, beliefs, permissions, or memory writes, depending on the attack modality. Non-code compositional harm (e.g., covert natural-language reassembly) remains an open problem.

Implications and Future Directions

• Security and monitor architecture: The observation window becomes an explicit security parameter; local safety guarantees do not imply global safety when harm is compositional.
• Benchmarking and evaluation: Evaluations must account for compositionality and representation boundaries, avoiding metrics that reward detection of superficial or residual cues.
• Open research questions: The main unsolved defensive problem is representation discovery—automatically identifying the context where the attack payload is exposed, especially for non-code harms. Adaptive, stateful, and cross-agent monitors may be required.
• Deployment challenges: Oracle-free, attack-training-free, and codec-unaware blocking remains out of reach; current successful blocking relies on knowledge of the encoding family.

Conclusion

This paper delineates the observability boundary of local monitors in multi-agent LLM systems, providing formal justification and rigorous empirical evidence that distributed backdoor attacks are undetectable by local monitors when fragments are locally benign and statistically indistinguishable. Detection and blocking only recover when monitors reach and inspect the assembled representation in which harm emerges; merely strengthening local classifiers or aggregating more fragments is insufficient. The implications extend to monitor design, benchmarking, and system security, foregrounding representation discovery as the primary challenge for detecting compositional harm in AI agent workflows (2607.11751).