Agentic HRP (ORCHID) System Overview

• Agentic HRP (ORCHID) is a modular on-premise AI system that classifies High-Risk Property using retrieval-augmented generation and human-in-the-loop oversight.
• It employs specialized agents for retrieval, description refinement, classification, validation, and feedback logging to comply with evolving export-control policies.
• Empirical evaluations show that ORCHID improves classification accuracy and traceability by integrating evidence-based reasoning with rigorous audit trails.

Agentic HRP (ORCHID) refers to the specialized artificial intelligence system "Orchestrated Retrieval‐Augmented Classification with Human‐in‐the‐Loop Intelligent Decision‐Making" (ORCHID), designed for the classification of High‐Risk Property (HRP) within compliance workflows at sites such as those operated by the U.S. Department of Energy. ORCHID extends retrieval-augmented generation (RAG) paradigms by embedding evidence-based reasoning, rigorous human oversight, and end-to-end auditability directly into classification processes governed by complex, evolving export-control policies (Mahbub et al., 7 Nov 2025).

1. System Architecture and Agentic Composition

ORCHID is architected as a modular, on-premise platform comprising five specialized agents coordinated by an Orchestrator. Inter-agent communication operates over a local message bus, employing an Agent-to-Agent (A2A) schema wherein each message encapsulates item metadata, context, citation links, proposed decisions, validator outputs, and provenance attributes including model versioning, timestamps, and configuration hashes.

The agentic modules are functionally delineated as follows:

• Retrieval Agent (IR): Accepts structured item inputs (manufacturer, name, model, user-supplied description) and formulates hybrid queries using both BM25 (lexical) and vector-based (semantic) search strategies. Through the Model Context Protocol (MCP), the IR retrieves top-K policy text snippets, each annotated with a section identifier and retrieval score.
• Description Refiner (DR): Processes and sanitizes noisy or incomplete user descriptions by invoking a local summarization tool (MCP‐Summary) or prompting the user for clarification, returning a refined description for downstream use.
• HRP Classifier (HRP): Constructs input prompts by assembling item metadata and citation-retrieved policy snippets, and proposes policy-constrained decisions (label, control category, confidence score, chain-of-thought rationale) via the local LLM using MCP.
• Validator (VR): Evaluates classifier outputs against retrieved evidence by assessing coverage sufficiency and policy snippet conflicts; outputs a verdict in {AGREE, REVIEW, CONFLICT}, with non-agree verdicts escalating to Subject Matter Expert (SME) review.
• Feedback Logger (FL): Records SME accept/override feedback with rationales as append-only audit entries, anchoring each to item and communication history.

All agent tool invocations remain local, facilitated by MCP adapters supporting JSON input/output. No external egress or cloud connectivity is permitted, confining the system entirely to on-premise operation.

2. Retrieval-Augmented Generation and Evidence Grounding

The retrieval-augmented classification process grounds every decision in formally documented policy text. The canonical policy corpus comprises eCFR USML (22 CFR §121), NRC (10 CFR §110), CCL (15 CFR §774), and EAR99 guidance, individually partitioned into identifiable textual passages and embedded using mxbai-embed-large-v1.

Hybrid indexing integrates BM25 lexical screening with semantic vector representations, merging results through Reciprocal Rank Fusion (RRF). IR-generated queries—constructed from structured and refined item descriptions—yield a ranked set of K policy excerpts. A "snippet packer" algorithm identifies the minimal subset necessary to collectively address all salient attributes for HRP decisions, minimizing input prompt length while ensuring evidence coverage.

Classifier input prompts strictly limit the model to referencing only on-policy, cited text, enforcing a hard constraint on the provenance of each classification outcome.

3. Decision Logic and Human-in-the-Loop Oversight

The HRP agent’s propositional output is structured as:

• Label ∈ {USML, NRC, CCL, EAR99}
• Confidence ∈ [0,1], computed from softmax-normalized model logits
• Explicit chain-of-thought reasoning with numbered citations

A configurable confidence threshold $\tau$ controls the gating mechanism:

• If classifier confidence ≥ $\tau$ and VR verdict = AGREE: the decision is automatically accepted.
• Otherwise (confidence < $\tau$ or VR verdict ∈ {REVIEW, CONFLICT}): the case is escalated to the SME queue.

SME reviewers accept or override final decisions, supplying a one-line rationale. FL logs every interaction with complete provenance, constructing a historical feedback corpus that may be utilized for system retraining or regulatory rule-tuning but is deliberately excluded from live inference to maintain audit integrity.

4. Auditability, Transparency, and Provenance

ORCHID enforces an auditable “Item→Evidence→Decision” life cycle, capturing all intermediate artifacts:

1. Item submission event, documented via a run-card including session ID, configuration hash, model/software versions, and timestamp.
2. Retrieval queries, returned snippet identifiers, and associated retrieval scores.
3. Prompt construction steps, including the template and all populated fields.
4. Model output serialization: JSON encapsulating label, confidence, citations, and reasoning trace.
5. Validator verdict (AGREE/REVIEW/CONFLICT).
6. SME feedback, including the rationale for overrides.

Artifacts are appended into audit bundles, which may be exported in JSON or PDF formats. Exported reports at batch or item level encode all versioning information critical for deterministic classification replay and forensic verification. The following pseudocode is provided for audit exportation:

function export_audit_bundle(session_id): bundle = {} bundle["run_card"] = load_run_card(session_id) bundle["events"] = load_all_A2A_messages(session_id) bundle["prompts"] = extract_prompts(bundle["events"]) bundle["model_outputs"] = extract_model_outputs(bundle["events"]) bundle["SME_feedback"] = load_FEEDBACK_logs(session_id) write_json(bundle, "audit_bundle_"+session_id+".json") write_pdf(render_report(bundle), "audit_report_"+session_id+".pdf")

Audit artifacts are immutable, append-only, and cryptographically tie the decision trail to both process and data provenance.

5. Empirical Performance and Comparative Evaluation

ORCHID was evaluated on real-world HRP cases, benchmarking against a non-agentic baseline consisting of direct, single-stage LLM classification absent retrieval or validation. Key empirical results are:

• Class-wise and overall accuracy:

USML	NRC	CCL	EAR99	Weighted Avg	Binary Acc
88%	90%	56%	40%	63.12%	70.37%

• Compared to the baseline LLM (≈ 55% weighted accuracy, 62% binary), ORCHID’s pipeline showed a +8 percentage point gain in weighted accuracy and +8.4 percentage points in binary high/low-risk recall.
• 100% of accepted ORCHID decisions included ≥2 on-policy citations, versus 35% for the baseline.
• F1 scores improved by class: USML (0.82→0.86), NRC (0.88→0.90), CCL (0.45→0.56).

These findings support a measurable improvement in both accuracy and compliance traceability through agentic workflow, RAG evidence provision, and human-in-the-loop validation.

6. Compliance, Security, and Operational Considerations

ORCHID’s design principles reflect strict auditability and policy compliance requirements: full operation is local and on-premise, precluding all data egress to the cloud or external APIs. The system logs every action and rationale—a requirement for transparency in U.S. DOE export-control workflows. Policy versioning and configuration hashes enable end-to-end reproducibility for every classification event.

This suggests that agentic HRP systems using structured RAG pipelines with immutable audit trails may serve as reference models for AI-enabled governance in other high-stakes or regulated settings.