Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming
Abstract: The fast growth of open-source AI infrastructure, from model serving engines and agent platforms to the Model Context Protocol (MCP) ecosystem and the LLMs themselves, has outpaced the security tooling available to defend it. We present AI-Infra-Guard, an open-source framework that organizes AI red teaming around a single observation: the attack surface of an AI agent is stratified across layers (infrastructure, protocol/tool, agent behavior, and model), and no single detection paradigm fits all of them. The framework therefore matches a paradigm to each layer, from deterministic rule matching over 75+ AI components and 1{,}400+ vulnerability rules, through LLM-driven agentic auditing of MCP servers and agent-skill packages and multi-turn black-box agent red teaming, to a jailbreak harness with 26+ attack operators over sixteen datasets. To our knowledge it is the only open-source framework to span all of these, including supply-chain auditing of the agent skills that increasingly extend AI agents. We release AI-Infra-Guard as open source so that \emph{layer-paradigm matching} can serve as a practical foundation for agent security and a shared base for the community to build on.
Summary
- The paper introduces AI-Infra-Guard, an open-source framework that aligns detection techniques with four distinct security layers.
- It employs deterministic rule matching, LLM-driven semantic auditing, multi-turn adversarial red teaming, and statistical evaluation.
- Empirical results demonstrate high F1 scores and recall, effectively exposing vulnerabilities across infrastructure, protocol, behavior, and model layers.
Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming
Introduction and Motivation
The rapid proliferation of open-source AI infrastructure—from model-serving engines and agent orchestration platforms to the Model Context Protocol (MCP) and the underlying LLMs—has resulted in a significantly expanded and stratified attack surface. Traditional security assessment tools, built around established catalogues and regular versioning conventions, fail to account for the irregularities and novel risks inherent to modern AI systems. The landscape is further complicated by the co-existence of four distinct security layers: infrastructure, protocol/tool, agent behavior, and model alignment. Each requires a fundamentally different detection paradigm. "Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming" (2606.31227) introduces AI-Infra-Guard, an open-source framework that operationalizes the principle of layer-paradigm matching. The system integrates deterministic rule matching, LLM-driven agentic auditing, multi-turn adversarial red teaming, and statistical alignment robustness evaluation, effectively spanning all critical strata of the AI agent attack surface.
Layer-Paradigm Matching: A Theoretical Foundation
The central theoretical claim advanced in this work is that effective security assessment in AI systems demands matching each attack surface layer with a detection technique that yields sufficiently strong evidence for the relevant class of vulnerabilities. The framework distinguishes four classes of evidence—signature, semantic, behavioral, and statistical—and aligns each with a corresponding detection paradigm:
- Infrastructure: Deterministic rule matching for observable signatures and known vulnerabilities
- Protocol/Tool: LLM-driven semantic analysis for reasoning about code, tool metadata, and MCP-specific attack vectors
- Agent Behavior: Multi-turn, adversarial black-box red teaming to expose runtime-only vulnerabilities
- Model Alignment: Large-scale statistical attack enumeration with LLM-based judgment to evaluate alignment robustness
This assignment is grounded in two formal principles:
- Security Heterogeneity Principle: Classes of AI security risks arise from fundamentally different system properties, precluding evaluation by a single universal methodology.
- Evidence Sufficiency Principle: A security assessment procedure is valid only if it produces evidence sufficient to establish the security property at each layer.
Infrastructure Scanning: Deterministic Rule-Based Layer
Infrastructure scanning targets the identification of deployed AI components, their versions, and known exposures. The module implements a declarative, boolean-expression-based matching language for both component fingerprinting and vulnerability detection, decoupled from conventional regular-expression-based engines. The fingerprint corpus covers 75+ AI components with over 1,400 vulnerability rules, supporting non-semantic version normalization to address the spectrum of versioning inconsistencies observed in practice.
Upon component identification, the matching engine applies version predicates or infers findings when version data is unavailable. It explicitly classifies findings into three confidence tiers: verified (evidence-based), version-matched (advisory-driven), and inferred (identity-only). The scan pipeline is highly parallelized and disk-backed, enabling coverage of large, potentially internet-scale target sets without sacrificing resource efficiency.
Figure 1: The infrastructure-scanning pipeline, combining fingerprint analysis, vulnerability matching, and agentic visual/risk assessment, culminating in a structured and actionable security report.
Protocol/Tool Layer: LLM-Driven MCP and Skill Auditing
The MCP auditing module addresses the inadequacy of signature-based detection for protocol-level vulnerabilities, such as command injection, credential theft, tool poisoning, and tool shadowing. AI-Infra-Guard deploys an agentic harness, leveraging LLM-as-auditor within bounded, multi-stage pipelines. The harness is equipped with specialized tools (filesystem, shell/grep, MCP interaction agents), dynamic context management, and role-based model routing. The detection corpus is encoded in natural-language Prompt-as-Rule specifications, with explicit inclusion and exclusion criteria designed to minimize both false positives and over-reporting, reflecting best practices from the OWASP MCP Top 10.
The system supports both static (white-box) and dynamic (black-box) MCP auditing. Static analysis enables direct reasoning over source; dynamic analysis confines itself to tool metadata and runtime interaction, emphasizing output-centric evidence validation, especially for tool poisoning and indirect prompt injection.
Figure 2: The MCP-auditing pipeline, illustrating the sequential reasoning, interaction, and validation carried out across static and dynamic assessment modes.
Skill auditing extends this paradigm to the AI agent supply chain. Skills—modular capability packages—introduce new supply-chain risks, including manifest tampering and privilege escalation. The agentic pipeline decomposes skill artifacts, applies lightweight static risk retrieval, and employs LLM-based contextual analysis within a controlled environment. The integration of external threat intelligence and iterative badcase-driven improvement loops further enhances the precision and adaptability of detection.
Agent Behavior: Multi-Turn Black-Box Red Teaming
Assessment of agent behavior necessitates adversarial multi-turn interaction. The framework abstracts attacks as skill manifests (advanced Prompt-as-Rule specifications), each operationalized as an attack playbook targeting specific risk families: data leakage, tool abuse, indirect injection, and authorization bypass. These manifest-constrained agents reason over target capabilities, adapt attack strategies dynamically, and enforce structured escalation ladders combined with cost-aware stop rules.
A salient design feature is the systematic anchoring of subjective LLM judgments with deterministic evidence. For canary token-based SSRF or indirect injection via marker tokens, confirmation moves beyond LLM opinion to verifiable output, greatly increasing report actionability.
Model-Layer Jailbreak Evaluation: Large-Scale Statistical Red Teaming
The model layer is assessed through comprehensive jailbreak evaluation, framed as a statistical robustness question. AI-Infra-Guard integrates a composable harness unifying a wide spectrum of attack operators (single-turn encoding, role-play, context poisoning, multi-turn crescendo/tree-search strategies) and red-teaming datasets (7,000 harmful prompts across 16 curated corpora). Modular plugin support and customizable LLM-as-judge metrics expand the system's range and adaptability. The framework generates directly comparable safety scores and attack success rates across models and deployments.
Figure 3: The jailbreak-evaluation harness, demonstrating the synergy between systematic attack operator application and rigorous model outcome judgment.
System Architecture and Integration
The detection modules are orchestrated through a distributed server-agent architecture, built for scalability and uniform dispatch of heterogeneous, resource-heterogeneous tasks. Workers (in Go and Python) execute scans and audits, streaming structured events and intermediate results to clients via WebSocket/SSE channels. The centralized knowledge base (fingerprints, CVEs, benchmarks) is used by all modules; rule bases and plugins are remotely updatable. Modular delivery via CLI, web UI, or agent skill ensures broad ecosystem integration.
Figure 4: The distributed server-agent architecture connecting user interfaces, core detection engines, agent modules, and centralized knowledge resources.
Empirical Results and Claims
- The skill supply chain module, evaluated on SkillTrustBench (5,520 cases across 9 threat categories), achieves a loose F1 score above 0.98 with leading models (e.g., Claude Opus 4.6, GLM 5.1), and recall near 1.0, confirming assessment effectiveness is model-dependent and that the detection harness exposes base-model limitations rather than adding bottlenecks.
- The framework is, to the authors' knowledge, the only open-source system that comprehensively spans infrastructure, protocol/tool, agent behavior, and model layers, including agent skill supply chain auditing, and matches each to its theoretically justified detection paradigm.
- Prompt-as-Rule, combined with explicit exclusion criteria and objective confirmation (canaries, markers), consistently improves both precision and recall, particularly when used in LLM-driven domains susceptible to over-reporting or prompt injection.
Broader Implications and Future Directions
This work establishes a formal and operational link between the spectrum of AI agent vulnerabilities and the evidentiary and methodological requirements for their assessment. Practically, the system models how heterogeneous, evolving agent ecosystems can be defended through extensible, community-driven knowledge bases and audit modules. Theoretical contributions include formalizing the evidence classes underlying security findings and setting a precedent for layer-adaptive security assessment.
The modular, compositional approach to attack operator and metric plugin management in the model-layer component enables rapid incorporation of emerging jailbreak techniques and benchmarks. Future work may focus on large-scale measurement of real-world agent exposures, cross-layer detection integration (feeding infrastructure/model findings into behavioral probes), and continuous expansion of rule and benchmark corpora.
Conclusion
"Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming" delivers a rigorously structured, technically diverse framework for agent security assessment. Through systematic application of layer-paradigm matching, it aligns detection procedures to the inherent nature of vulnerabilities at each layer, ensuring both pragmatic effectiveness and theoretical soundness. The adoption of Prompt-as-Rule, objective confirmation, and scanner self-defense patterns serves as a blueprint for future security tooling in complex AI ecosystems. This work sets a new standard for comprehensive, evidence-driven AI agent security auditing and defines an extensible architecture for the evolving threat landscape.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper introduces AI-Infra-Guard, a free (open‑source) toolkit to help keep AI “agents” safe. AI agents are software assistants that can read files, use tools, browse data, and talk to people. Because they connect to many systems, they can be attacked in different ways. The authors’ main idea is simple: think of an AI agent like a castle with several walls. Each wall needs a different kind of guard. So they build one framework that checks security at four layers—machines and servers, the tools and protocols the agent uses, the agent’s behavior, and the AI model itself—using a different, best‑suited method for each layer.
What questions were they trying to answer?
In everyday terms, they asked:
- How do we spot risks in the new AI software stack when old security tools don’t recognize these new parts?
- How can we test different kinds of weaknesses: misconfigured servers, risky tool code, agents that can be tricked during conversation, and models that can be “jailbroken”?
- Can we organize all this into one system so teams can scan everything in a consistent way?
How did they do it? (Explained with analogies)
The authors match each “layer” of the AI stack with a fitting “security test,” like using the right tool for each job.
- Layer 1: Infrastructure (the machines and services)
- What it is: Things like model servers (e.g., Ollama, vLLM), web consoles, and ML platforms that run on your network.
- How they test: Fast, rule‑based checks—like a supermarket barcode scanner. They look for known signs (fingerprints) that identify software and versions, then compare against a large set of known issues. They even handle messy version names (like “2.3.dev” or “latest”) so they don’t get confused.
- Why this fits: It’s quick, consistent, and great for known problems.
- Layer 2: Protocols and tools (the agent’s “hands”)
- What it is: MCP servers and “skills” that tell the agent how to read files, query databases, or run commands.
- How they test: An LLM‑guided “auditor” agent reads code and descriptions the way a human reviewer would. It looks for dangerous patterns and misleading tool descriptions (for example, a tool that says it reads files but secretly sends data out). This is called “Prompt‑as‑Rule”: instead of fixed code rules, the system gives the AI clear, step‑by‑step instructions (prompts) on what to look for and what to ignore.
- Why this fits: Some problems can’t be caught with simple rules—you have to understand what the code and descriptions mean.
- Layer 3: Agent behavior (how the agent acts in conversation)
- What it is: The agent you chat with—can it be tricked into revealing secrets or misusing tools?
- How they test: Black‑box “red teaming,” which means they talk to the agent like a clever attacker would, across multiple turns, to see if it slips up. They use safety “stop rules,” canary tokens (fake secrets to detect leaks), and cost controls so testing is safe and affordable.
- Why this fits: Some issues only show up during live interaction, not by reading code.
- Layer 4: The model itself (the brain)
- What it is: The LLM’s own safeguards—can it resist being “jailbroken” by tricky prompts?
- How they test: A “jailbreak harness” that tries many kinds of attacks from multiple datasets, then uses a model‑as‑judge to decide if the attack worked.
- Why this fits: Robustness here is statistical—you need lots of tries to know how often the model fails.
Two extra ideas that make this practical:
- Context management: The auditor summarizes earlier findings as it goes, so it can handle big codebases without forgetting the important parts.
- Model routing: It uses stronger, slower models for hard judgments and faster, cheaper ones for simple tasks to keep costs down.
Simple definitions:
- MCP (Model Context Protocol): A standard that lets AI agents discover and use external tools (like “read file” or “query database”).
- Red teaming: Pretending to be an attacker to test defenses.
- Prompt injection: Hiding instructions inside content to trick an AI into doing the wrong thing.
- Jailbreak: A prompt that gets a model to ignore its safety rules.
What did they build or find, and why is it important?
- A unified framework that covers all four layers in one place. This is rare: most tools focus on just one layer.
- A big rule set for infrastructure scanning: 75+ AI components, 100+ fingerprint rules, and 1,400+ vulnerability rules, including ways to handle messy versioning and directly confirm unsafe exposures (like accidentally exposed config files).
- An LLM‑powered auditor for MCP servers and “skills” with a Prompt‑as‑Rule approach, plus sandboxing and defenses against being tricked by malicious content during the scan.
- A practical agent red‑teaming setup that balances coverage and cost, using multi‑turn strategies, escalation ladders, and canary tokens.
- A jailbreak evaluation harness with 26+ attack methods across 16 datasets, plus a “model‑as‑judge” to decide success.
- One architecture and one workflow: every module accepts a target, gathers info, runs checks, and outputs structured findings with severity and fixes. That makes it easier for teams to use and for the community to extend.
Why this matters:
- Old security scanners often miss AI‑specific problems (like tool poisoning or prompt injection). This framework directly targets them.
- By using the “right test for the right layer,” it reduces false alarms where possible and digs deep where needed.
- It’s open source, so others can add new rules and attacks as the AI ecosystem evolves.
What could this change in the real world?
- Safer deployments for startups and small teams: Many people run AI services without advanced security; this gives them a practical toolkit.
- Better protection for businesses: Stops expensive compute abuse, helps prevent data and credential leaks, and reduces the chance of agents being tricked.
- Stronger AI ecosystems: Auditing the “supply chain” of tools and skills helps prevent hidden risks from spreading.
- A shared foundation: Because it’s open and layered, researchers and engineers can build on it together, improving defenses as new threats appear.
In short, the paper’s big idea is that AI systems need different kinds of checks at different layers, and AI-Infra-Guard turns that idea into a working, community‑friendly toolkit.
Knowledge Gaps
Below is a consolidated list of concrete knowledge gaps, limitations, and open questions that remain unresolved in the paper. Each point is framed to guide future research and engineering work.
- Empirical validation is missing: no quantitative evaluation (e.g., precision/recall, false-positive/false-negative rates, inter-rater agreement with human auditors) for any module, despite the “precision stratification” claims for infrastructure findings.
- Benchmarking absent: no throughput/latency, scalability, or resource-use benchmarks for large target sets (e.g., million-IP CIDRs), and no comparison to existing scanners or red-teaming harnesses.
- Rule corpus coverage limits: only 75 components and 1,443 vulnerability rules are covered; the paper does not quantify ecosystem coverage or identify priority gaps (e.g., popular agent platforms, gateways, RAG stacks, or emerging MCP servers).
- Update and governance of rules: no process for curating, validating, de-duplicating, and de-conflicting rules contributed by the community; lack of a CI pipeline for rule testing and regression prevention.
- Version normalization correctness: heuristics mapping tags like
latest,rc,dev, and build numbers (e.g.,b7824) have no formal correctness guarantees or error bounds; no tests on multi-branch/non-monotonic version schemes. - Risk of “inferred findings”: empty-predicate rules that fire on identity alone can inflate false positives; the paper lacks a calibration method or triage protocol to balance recall vs. precision for such findings.
- Fingerprinting evasion: no strategy for resilient detection when services obfuscate headers/bodies, sit behind CDNs/WAFs, randomize content, or use gRPC/WebSocket-only interfaces; limited discussion of non-HTTP protocols.
- Auth-protected surfaces: framework primarily targets unauthenticated probing; no methods for authenticated scanning (e.g., safe credential handling, OAuth flows) or detection of misconfigurations only visible post-auth.
- Encrypted/behind-proxy deployments: no techniques for detecting components behind TLS termination at gateways, reverse proxies, or service meshes (e.g., Envoy, Istio) where direct fingerprints are suppressed.
- Infrastructure visual analysis: “dynamic page rendering, visual asset capture, and multimodal risk assessment” are mentioned but lack architectural detail, model choice, evaluation, or safety validation for the vision component.
- MCP static auditor ground truth: no evaluation on a labeled corpus of known vulnerable MCP servers/tools; absence of recall/precision metrics or comparisons with human code review.
- Dynamic MCP testing safety: unclear safeguards to prevent destructive side effects when calling live tools (e.g., file deletion, network exfiltration); missing sandboxing, permission-scoping, or dry-run mechanisms.
- Prompt-as-Rule specification: no formal schema, lifecycle, versioning, or linting/validation tooling for “detection criteria” prompts; unclear how to ensure portability across base models or prevent prompt drift.
- Prompt-injection resilience: although the auditor treats artifacts as untrusted, there is no adversarial evaluation quantifying resilience to indirect prompt injection (e.g., poisoned tool descriptions, readme files, or code comments).
- Context compaction fidelity: no study of information loss from conversation summarization (e.g., ablations showing how compaction thresholds affect detection quality or false negatives in large repositories).
- Model routing policy: the framework lacks a reproducible policy for choosing models per sub-task and no cost–accuracy analysis across model choices or providers; no fallback strategy under provider failures.
- Tool sandbox completeness:
read_filepath sandboxing is described, but dynamic tool invocation and shell calls (even for grep) may expand the attack surface of the scanner itself; no formal threat model or isolation guarantees. - Safe handling of secrets: scanning may expose private code and credentials to external model providers; data residency, redaction, encryption, and retention policies are unspecified.
- Legal/ethical scanning: no guidance on safe/authorized use (e.g., scope restrictions, consent records, non-destructive testing policies) when probing third-party endpoints or live MCP servers.
- Agent red-teaming coverage: the (later) agent module is referenced but no concrete coverage model, attack space enumeration, or stopping criteria effectiveness studies are provided (e.g., how escalation ladders/canaries impact missed vulnerabilities).
- Canary-token reliability: effectiveness and false-positive/false-negative rates of canary-based verification in real agent environments remain unquantified.
- Model-level jailbreak harness validity: reliance on LLM-as-judge lacks calibration (e.g., judge bias, self-judging issues, cross-model robustness, consistency across runs, and statistical significance reporting).
- Dataset representativeness: the 16 jailbreak datasets and 26+ operators lack description of domain coverage (e.g., multilingual, multimodal, task diversity) and do not assess transfer to unseen attacks or non-text modalities.
- Cross-layer correlation: no mechanism to fuse findings across layers into attack paths (e.g., infra exposure → poisoned skill → agent misuse → model jailbreak) for end-to-end risk quantification and prioritization.
- Scoring calibration: aggregate security score derivation is not specified; no validation that scores correlate with real business risk or incident likelihood/impact across layers.
- Remediation guidance quality: automatically generated fix suggestions are not evaluated for correctness or feasibility; risk of hallucinated or unsafe remediation instructions is unaddressed.
- Continuous monitoring: no scheduling/version-tracking strategy for continuous scanning and drift detection (e.g., detecting when rules, versions, or behaviors change between runs).
- SBOM/SLSA alignment: the framework does not integrate with AI-specific SBOMs (models, prompts, tools) or supply chain attestations (e.g., SLSA) to enhance coverage and provenance trust.
- Beyond MCP: limited discussion on scanning other tool invocation ecosystems (e.g., OpenAI tool spec, LangChain tools, custom RPC), or skill formats outside the paper’s focus.
- Non-English/localized UIs: fingerprinting and rule logic appear English-centric; detection performance on localized consoles and messages is unexplored.
- Air-gapped/offline deployments: no guidance on running with local models and offline rule sets, or on updating corpora in restricted environments.
- Resilience to rate limiting: the cost-bounded and rate-limited approaches are not empirically validated against provider throttling behaviors and failure modes.
- Worker/server security: the orchestration stack adds its own attack surface (WebSocket channels, task dispatch, plugin subprocesses); no penetration testing or hardening guidance is provided for AI-Infra-Guard itself.
- Evasion via content negotiation: fingerprinting may miss variant responses (e.g., language, mobile, gzip, content negotiation); no strategy to probe multiple variants for robust identification.
- Dependency-level vulnerabilities: infra matching focuses on component versions but does not analyze internal dependencies (e.g., vulnerable libraries within MCP servers or agent skills).
- Non-HTTP protocols and GPU services: limited strategy for fingerprinting model servers exposing gRPC-only interfaces or specialized GPU services and job schedulers beyond HTTP endpoints.
- Evaluation reproducibility: the lack of released evaluation scripts, seed corpora, or configuration snapshots hinders independent replication of claimed design benefits.
- Economic analysis: no clear costing model for LLM usage per scan/task and no ROI analysis comparing added detection power versus compute spend for each layer.
- Extensibility boundaries: unclear how far Prompt-as-Rule and the harness generalize to multimodal tools, audio/vision agents, or non-textual MCP resources; no roadmap for these modalities.
Practical Applications
Below is a concise mapping from the paper’s findings and methods to real-world, practical uses. Each item names the target users/sectors, the workflow or product that could emerge, and key assumptions/dependencies that affect feasibility.
Immediate Applications
Industry and Enterprise (Software, Cloud, Finance, Healthcare, Robotics, Energy, E‑commerce)
- Deployable AI infra exposure scanning (M1: Infra-Scan)
- Use case: Security/SRE teams scan networks for self-hosted AI components (e.g., Ollama, vLLM, Dify, ComfyUI, MLflow, Kubeflow, Ray), identify versions, and detect misconfigurations (e.g., unauthenticated endpoints, leaked configs).
- Tools/workflows: Integrate the Go-based scanner and rule corpus (fingerprints + vulnerability rules + version normalization) into scheduled scans; export security scores and inventories to SIEM.
- Dependencies: Permission to scan, network reachability, maintenance of the 75+ component signatures and 1,400+ rules, accuracy of version normalization for non-semver tags (e.g., b7824, .dev, rc).
- CI/CD “AI security gate” for agent platforms and MCP servers (M2 static)
- Use case: Add LLM-driven static auditing of MCP servers and agent-skill packages to release pipelines to block tool poisoning, command injection, credential mishandling.
- Tools/workflows: A “Security Check” step that runs the agentic harness (Prompt-as-Rule criteria, sandboxed read_file, grep), reports structured findings with severities, and remediation tips.
- Dependencies: Access to source repos; stable LLM access (OpenAI-compatible gateway) and cost limits; continuous prompt/rule updates; sandboxing for repo reads.
- Runtime agent abuse testing pre-release (M3 black-box)
- Use case: Pre-deployment red teaming of chatbots/assistants built on platforms like Dify/Coze to probe prompt leakage, confused-deputy tool abuse, SSRF/path traversal.
- Tools/workflows: Multi-turn, cost-bounded dialogues with escalation ladders, stop rules, and canary tokens; automated reports with evidence and reproducibility.
- Dependencies: Staging endpoints, rate limits and API quotas, canary infra, model/tooling for LLM-as-controller; legal approval for probing.
- Model jailbreak/regression gating (M4 harness)
- Use case: Release gating for foundation/finetuned models by enumerating 26+ attack operators across multiple datasets with LLM-as-judge; prevent alignment regressions.
- Tools/workflows: Add a “Prompt-Security” step in Evals; baseline scores, diffs per release; per-category failure alerts (e.g., self-harm, weapons).
- Dependencies: Reliability of LLM-as-judge, dataset representativeness, compute budget for many adversarial trials; model/provider rate limits.
- MSP/SOC service add-on: “AI Surface Scan”
- Use case: Managed security providers bundle AI-Infra-Guard to inventory customer AI assets, track exposures, and provide monthly remediation plans.
- Tools/workflows: Central scheduler + WebSocket worker pool; dashboards fed by structured findings and scores per tenant; ticketing integrations.
- Dependencies: Multi-tenant isolation; credential and data handling policies; ops capacity to triage inferred vs verified findings.
- Procurement/vendor due diligence for MCP tools/skills
- Use case: Security teams evaluate third-party MCP servers and agent skills prior to adoption using static and dynamic harness pipelines.
- Tools/workflows: A checklist backed by Prompt-as-Rule and runtime tests; sign-off requires “no critical” findings; attach scorecards to procurement records.
- Dependencies: Access to repos or demo endpoints, standardized reporting; consistent definitions of severity; time-boxed auditing budgets.
- Marketplace pre-submission scanning (skills/MCP)
- Use case: Marketplaces (enterprise internal or public) auto-scan submissions for tool poisoning, over-privilege, embedded prompt injection.
- Tools/workflows: Server-side audit harness; block submissions with high-severity findings; provide fix guidance.
- Dependencies: Throughput vs cost; appeals/re-review process; stable rule sets; handling of dynamic code-loading patterns.
- Cloud cost and credential-loss prevention
- Use case: Catch unauthenticated GPU endpoints and leaked API keys/config files before abuse.
- Tools/workflows: Verified findings for config disclosure (.env, .cursor/mcp.json), default unauthenticated access flags; integrate with cloud alerting and auto-quarantine.
- Dependencies: Read access where configs are publicly reachable; clear remediation playbooks; incident response hooks.
Academia and Research
- Reproducible multi-layer AI security benchmarks
- Use case: Evaluate new LLM auditors, detection prompts, or attack operators within a shared, open, four-layer harness.
- Tools/workflows: Swap base models; compare Prompt-as-Rule variants; log structured evidence classes (signature, semantic, behavioral, statistical).
- Dependencies: Access to models; standardized datasets; reporting standards for precision/recall and cost.
- Course labs and capstones in “AI Systems Security”
- Use case: Hands-on labs for students to scan infra, audit MCP code, red-team agents, and evaluate model robustness.
- Tools/workflows: Pre-built lab targets; controlled “bad” repos; safe sandboxes.
- Dependencies: Classroom compute budgets; safe target provisioning; ethics approvals.
Policy, Risk, and Compliance
- Internal AI deployment policy with layer-adaptive checks
- Use case: CISOs formalize minimum controls per layer (infra rule scans; MCP/skill audits; runtime agent tests; model jailbreak checks) as a standard.
- Tools/workflows: Policy templates; control mapping to NIST AI RMF/ISO/sector-specific frameworks; automated proof via scan artifacts.
- Dependencies: Executive sponsorship; change management; trained owners for each layer.
- MLOps/SecOps control evidence for audits
- Use case: Demonstrate due diligence for HIPAA, SOX, PCI, or EU AI Act readiness with structured findings and remediation trails.
- Tools/workflows: Retain signed reports, risk scores, and fixes; map to evidence classes per control.
- Dependencies: Auditor acceptance of LLM-based evidence; documentation rigor.
Individual Developers and Small Teams (Daily Life)
- Self-hosted AI stack “health check”
- Use case: Hobbyists/startups running Ollama, ComfyUI, or LangFlow scan their public endpoints for exposures and fix misconfigs.
- Tools/workflows: One-click Docker compose with Infra-Scan + report; link to remediation guides.
- Dependencies: Willingness to run scanners; basic ops literacy.
- Safer plugin/tool authoring
- Use case: MCP tool developers run static Prompt-as-Rule audits locally before publishing.
- Tools/workflows: CLI “aig audit” for repos; pre-commit hook; CI badge.
- Dependencies: Internet/model access (or local model); responsiveness to findings.
Long-Term Applications
Industry and Ecosystem
- Certification and scorecards for AI agents, MCP servers, and skills
- Use case: Third-party attestations (akin to SOC2/SLSA) require passing multi-layer scans; buyers demand vendor scorecards.
- Tools/products: “AI Agent Security Mark”; machine-readable attestation artifacts; periodic surveillance audits.
- Dependencies: Industry consensus; governance body; consistent baselines across LLMs and targets.
- Cloud-native “AI Security Posture Management” (AI‑SPM)
- Use case: CSPs integrate layer-paradigm scanning into managed AI services to continuously inventory, scan, and auto-remediate misconfigurations.
- Tools/products: Posture graphs linking infra, tools, agents, models; policy-as-code for AI controls; auto-rollbacks/kill switches.
- Dependencies: Deep CSP integration; safe automation; customer segregation.
- Runtime agent sandboxes and privilege brokers
- Use case: Enforce least-privilege and real-time containment for agent tool invocations informed by findings (e.g., flag high-risk tools, require step-up auth).
- Tools/products: Policy engine + MCP-aware broker; risk scoring per tool call; human-in-the-loop approvals for sensitive actions.
- Dependencies: Changes to agent/tool runtimes; developer adoption; UX for approvals.
- SBOM for AI agents and skills (“AI‑SBOM”)
- Use case: Standardized manifests that enumerate model versions, tools, skills, permissions, and known exposures; feed into scanners and auditors.
- Tools/products: AI‑SBOM spec and generators; ingestion into Infra-Guard; diff and drift detection.
- Dependencies: Standardization across vendors; package metadata norms for skills.
- Continual red teaming with self-play and adaptive operators
- Use case: Operator libraries evolve online and co-adapt to target defenses; continuous learning loops improve detection/defense.
- Tools/products: Feedback-driven attacker/defender pipelines; automated regression discovery.
- Dependencies: Guardrails against overfitting/false positives; compute budgets; privacy constraints.
Academia and Standards
- Formalization of evidence classes and evaluation protocols
- Use case: Community standards that define signature/semantic/behavioral/statistical evidence, metrics, and inter-rater reliability for LLM-as-judge.
- Tools/products: Public leaderboards; gold-labeled corpora for MCP/agent vulnerabilities; cross-lab reproducibility packs.
- Dependencies: Shared datasets; funding for curation; consensus on measurement error.
- Research on indirect prompt-injection–resilient auditors
- Use case: Robust harness designs that withstand adversarial artifacts (e.g., tool descriptions trying to steer the scanner).
- Tools/products: Benchmarks; certified-safe prompt patterns; defensive context management strategies.
- Dependencies: Access to diverse malicious artifacts; collaboration with marketplaces.
- Safer-by-design agent frameworks
- Use case: Agent platforms integrate built-in checks (capability scoping, privilege separation, audit logs) guided by audit findings.
- Tools/products: Default “secure templates”; code generators that enforce safe tool schemas and minimal privileges.
- Dependencies: Maintainer buy-in; backward compatibility.
Policy and Regulation
- Sector-specific baseline controls for AI agents
- Use case: Regulators publish required controls per layer for healthcare, finance, and critical infrastructure (e.g., runtime behavioral tests for PHI leakage).
- Tools/products: Compliance profiles; conformance testing harnesses derived from Infra-Guard.
- Dependencies: Policy development cycles; harmonization across jurisdictions.
- Vulnerability disclosure taxonomy for AI infra and agents
- Use case: Augment CVE/CWE with new classes (tool poisoning, tool shadowing, agent confused-deputy) and versioning norms for non-semver.
- Tools/products: New CWE entries; NVD updates; industry advisories templates.
- Dependencies: Coordination with MITRE/NVD; vendor participation.
Individuals and Small Teams
- Community-maintained “Prompt-as-Rule” registries
- Use case: Shared, vetted natural-language detection criteria for common vulnerabilities across languages/frameworks.
- Tools/products: Curated rule hubs; versioned rule packages; quality gates and voting.
- Dependencies: Moderation; legal concerns for sharing exploit patterns.
- Local/offline auditing stacks
- Use case: Privacy-sensitive teams run all modules with local models and on-prem workers.
- Tools/products: Model adapters, quantized LLMs for code/analysis; air-gapped runners.
- Dependencies: Model capability vs compute constraints; maintenance overhead.
Notes on cross-cutting assumptions/dependencies:
- LLM quality and variability: Findings in semantic/behavioral/statistical layers depend on the base model’s reasoning and the LLM-as-judge reliability.
- Cost and rate limits: Multi-turn red teaming and large-scale attack enumeration require budgeting tokens/requests.
- Legal/ethical constraints: Scanning must respect authorization boundaries; black-box testing of live endpoints requires explicit permission.
- Corpus maintenance: Fingerprint/vulnerability rules and Prompt-as-Rule criteria must evolve with fast-moving AI stacks.
- Dataset representativeness: Jailbreak and agent attack datasets may not fully cover emerging threat modalities; periodic updates are needed.
Glossary
- alignment robustness: The degree to which a model resists unsafe behavior under adversarial prompts and transformations. "alignment robustness"
- agent-skill supply chain: The ecosystem of packaged capabilities (skills) that agents install and rely on, which can introduce supply-chain risks. "agent-skill supply chain"
- agentic auditing: An LLM-driven, tool-using audit approach that reasons iteratively like an agent to analyze code and behavior. "LLM-driven agentic auditing"
- attack operators: Reusable adversarial transformations or tactics applied to prompts/models during red teaming. "26+ attack operators"
- attack surface: The collection of points or layers where an AI system can be attacked or abused. "the attack surface of an AI agent is stratified"
- attack-enumeration: Systematically generating and applying many attack variants to assess vulnerabilities. "attack-enumeration with model-based judgment"
- black-box red teaming: Adversarial testing that probes a system solely through its inputs/outputs without internal access. "multi-turn black-box agent red teaming"
- canary-token verification: Using planted, traceable tokens to validate data exfiltration or leakage during tests. "canary-token verification"
- Cataloguing gap: The lack of existing signatures or entries for new AI components in mainstream security databases. "Cataloguing gap."
- CIDR block: A notation for specifying IP address ranges used to define large scan targets. "a wide CIDR block"
- command injection: A vulnerability where untrusted input is interpreted as commands, enabling unintended execution. "command-injection flaw"
- confused deputy: An attack where an entity with authority is tricked into misusing its privileges on behalf of an attacker. "a confused deputy"
- context compaction: Summarizing earlier conversation history to stay within an LLM’s context limits while retaining key state. "context compaction"
- context window: The maximum token capacity an LLM can attend to in a single interaction. "context window"
- CVE: Common Vulnerabilities and Exposures; a standardized catalog of publicly known security vulnerabilities. "CVE matching"
- deterministic rule matching: Signature-based detection using fixed logic to identify components and vulnerabilities. "Deterministic rule matching."
- Evidence Sufficiency Principle: The idea that an assessment is valid only if it produces the kind of evidence necessary to prove the security claim. "Evidence Sufficiency Principle."
- gRPC endpoints: Remote procedure call interfaces using gRPC, commonly exposed by AI infrastructure. "gRPC endpoints"
- imperative escape hatch: Allowing procedural code to handle cases that resist declarative rules, increasing expressiveness. "An imperative escape hatch."
- indirect prompt injection: Malicious instructions embedded in content that an LLM processes, causing it to deviate from intended behavior. "indirect prompt injection"
- jailbreak-evaluation harness: A framework that applies many jailbreak attacks and judges success to assess model safety. "jailbreak-evaluation harness"
- Layer-Adaptive Assessment Principle: Matching each attack-surface layer with the least costly assessment capable of producing sufficient evidence. "Layer-Adaptive Assessment Principle."
- layer-paradigm matching: Aligning detection methods (rules, LLM auditing, red teaming) to the specific layer’s needs. "layer-paradigm matching"
- LLM-as-judge: Using a LLM to evaluate whether an attack attempt succeeded. "LLM-as-judge."
- LLM-driven agentic analysis: Employing LLMs with tools and iterative reasoning to analyze code and systems beyond fixed rules. "LLM-driven agentic analysis"
- Model Context Protocol (MCP): A protocol that lets models discover and invoke external tools via standardized descriptions. "Model Context Protocol (MCP)"
- OpenAI-compatible gateway: An API layer that exposes different models behind a uniform OpenAI-style interface. "OpenAI-compatible gateway"
- over-privilege: Granting a tool or skill more permissions than necessary, increasing blast radius if compromised. "over-privilege"
- path sandbox: A restriction that confines file operations to a designated directory to prevent traversal attacks. "path sandbox"
- path traversal: Exploiting file path handling to access files outside intended directories. "path traversal"
- pluggable base model: A configurable choice of underlying LLMs used by the auditing harness. "a pluggable base model"
- Prompt-as-Rule: Encoding detection logic as natural-language criteria applied by an LLM instead of fixed code patterns. "Prompt-as-Rule"
- range-intersection procedure: Combining multiple version constraints into the tightest consistent range for matching. "range-intersection procedure"
- reason-act loop: An iterative cycle where an LLM plans, invokes tools, observes results, and repeats to progress an audit. "reason-act loop"
- red teaming: Adversarial testing intended to discover vulnerabilities by thinking like an attacker. "AI red teaming"
- rug pulls: Tools or resources that change or disappear after adoption, undermining reliability/security. "rug pulls"
- semantic-version comparison logic: The ordering and comparison rules for semantic versions (e.g., MAJOR.MINOR.PATCH). "semantic-version comparison logic"
- server-agent architecture: A design where a central server coordinates distributed worker agents that perform scans. "server-agent architecture"
- server-side request forgery: Causing a server to make unintended network requests, potentially accessing internal services. "server-side request forgery"
- signature corpus: A curated set of fingerprints and rules used to identify components and known issues. "signature corpus"
- sized wait group: A concurrency control that caps the number of simultaneous tasks in the scanning engine. "a sized wait group"
- supply-chain auditing: Evaluating third-party packages or tools for risks like poisoning or over-privilege. "supply-chain auditing"
- system prompt: The hidden instructions or policy that govern an agent’s behavior, distinct from user input. "system prompt"
- tool poisoning: Crafting malicious tool descriptions or behavior to mislead or compromise an LLM-driven agent. "tool poisoning"
- tool shadowing: Registering deceptive tools that overshadow legitimate ones to hijack agent behavior. "tool shadowing"
- unauthenticated access: Exposing functionality without requiring credentials, enabling misuse or resource theft. "unauthenticated access"
- version-normalization procedure: Converting irregular version strings into a comparable format for matching. "version-normalization procedure"
- version predicate: A rule condition that specifies affected version ranges for a vulnerability. "version predicate"
- white-box auditing: Analysis that has access to source code and internals, as opposed to pure black-box probing. "static white-box auditing"
Collections
Sign up for free to add this paper to one or more collections.