CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents

Abstract: We introduce CFAgentBench, a reproducible, self-hostable environment and benchmark for autonomous construction-finance agents: a CFO/controller-class agent operating across the real software stack a US construction finance team runs - ERP, project management, email, documents, pay applications, payroll, certified payroll, lien waivers, and bank/treasury portals. It contains 1,014 machine-gradeable task specifications across 8 domains and 77 families, every family grounded in a real source; a self-validated subset of 40 tasks (54 with a project-management extension) is compiled into oracle-validated executable evaluators, the runnable suite reported here. Following WebArena, the benchmark runs on an executable environment rather than static traces: 35 mock applications (31 reconciled to one company book, plus 4 PM platforms) over 9 archetypes, each implementing a uniform self-hostable app contract, so every task is graded by functional correctness - a state diff plus forbidden-side-effect checks plus required-output regexes - with an LLM judge used only for reply quality, never as reward. A distinguishing principle is a money-movement guard: 278 instances embed a payment, payroll, e-signature, or e-filing step where the correct behavior is to stop and stage for human approval, and executing even the correct transaction fails the task. The public split (n=711) is sized for a 95% Wilson half-width of +/-4.1%; a private, contamination-protected split (n=303) is reserved for remote scoring. In a first three-model open-weight sweep (k=5), the strongest agent reaches pass^1 = 0.67 but only pass^5 = 0.38 - losing 43% of its successes when required to repeat them under temperature-0 decoding. The within-model pass^1 to pass^5 collapse and sharp per-domain heterogeneity are clear evidence that single-attempt accuracy overstates deployable construction-finance competence.

• The paper introduces CFAgentBench, a reproducible benchmark that rigorously evaluates autonomous construction-finance agents using money-movement safeguards and multi-tier reliability measures.
• The paper details a layered environment architecture across 35 applications and 8 domains, ensuring deterministic state transitions and reproducible testing conditions.
• The paper shows that single-attempt accuracy hides reliability collapse, emphasizing the need for consistent performance in high-consequence construction-finance workflows.

CFAgentBench: A Reproducible Benchmark for Autonomous Construction-Finance Agents

---

Motivation and Problem Formalization

CFAgentBench addresses the critical gap in evaluating autonomous agents for construction finance, a vertical characterized by job-cost-driven accounting, heterogeneity of systems, exception-heavy workflows, fragmented identity mappings, pervasive portal usage, and document-based ground truth. Unlike prior agent benchmarks that focus on consumer applications or standard tool-use domains, construction finance demands rigorous, functional state changes across ERP, project management, payroll, compliance, document repositories, and bank portals, all reconciled to a unified company book. The risk associated with incorrect financial actions in this domain mandates not just high accuracy but measurable reliability, with enforceable controls on money movement; single unauthorized payments or miscoded jobs are intolerable for controls-focused CFOs.

CFAgentBench operationalizes these requirements by constructing a reproducible and self-hostable environment with deterministic grading on functional correctness, extending concepts from WebArena (Zhou et al., 2023), SWE-bench (Jimenez et al., 2023), $\tau$-bench (Yao et al., 2024), AppWorld (Trivedi et al., 2024), and ToolBench/Gorilla (Qin et al., 2023, Patil et al., 2023). Its unique contribution is the explicit modeling and evaluation of money-movement guards and domain-specific system constraints.

---

Environment Architecture and Application Surface

The benchmark comprises 35 applications instantiated from 9 archetypes (ERP, Bank, Payroll, FieldTime, Project Management, Pay Application, Lien/Certified Payroll/Tax, Integration/Estimating, Docs/Office), each implementing a uniform six-capability app contract (seed, serve, snapshot, reset, state_diff, guarded_calls). This contract enforces determinism, reproducibility, and composability, enabling both in-process and HTTP-based deployments via parameterized Docker containers. Every app is seeded from a canonical company snapshot, and discrepancies across systems are documented and deliberately keyed. Money-movement endpoints (payments, payroll release, wire initiation, e-signature, e-filing) are strictly mocked, returning only staged, non-mutated states and ensuring no agent can pass via executing real transactions.

Tier-A (fully simulated), Tier-B (real-system Docker snapshots), and Tier-C (live read-only shims) provide layered fidelity, although all reported results leverage Tier-A. Each environment instance is reset, seeded, and snapshotted per task attempt, guaranteeing deterministic reproducibility and contamination control.

---

Dataset, Task Families, and Provenance

CFAgentBench v1 offers 1,014 machine-gradeable tasks across 8 domains (AP/Procurement, Reporting/Compliance, Project Accounting, Cross-system, Payroll/HR, Billing, Cash/Treasury, GL/Close) and 77 families. Instances capture realistic workflows, each grounded in primary sources (CFMA community threads, NAHB builder forums, "Finance at the Jobsite" podcast episodes, vendor documentation, customer emails/calls), with provenance recorded at the instance level.

Task schemas specify the systems involved, instruction text, governing policy references, defect/adversarial variants, expected/forbidden state diffs, money-movement guards, required output regexes, difficulty (equivalent human time), role persona, split partitioning, and contamination metadata. Oracle-validated executable evaluators are compiled for 40 tasks (54 with the PM extension), ensuring soundness of the assessment pipeline.

---

Evaluation Methodology

Grading prioritizes functional correctness of state transitions, aligning with the two-list control principle (expected and forbidden diffs). The layered grader checks:

• L1: Primary—expected state_diff
• Forbidden: Side-effect—no disallowed mutations
• Required outputs: Regex/string assertions on narrative replies
• L2: Structured call AST validation (where needed)
• L3: LLM judge for reply quality (never as reward signal)

The unique money-movement guard is enforced as a hard gate: any executed money movement (even when correct) fails the task; proper staging passes. Reliability is measured via $\mathrm{pass}^1$ (single attempt) and $\mathrm{pass}^k$ (k repeat passes), capturing the probability of consistent performance, not just capability.

Statistical power is calibrated by split sizes. The full public test set ($n=569$) can yield a 95% Wilson half-width of $\pm 4.1\%$ on accuracy; current experimental runs (n=40) have wider intervals ($\pm 0.15$).

---

Experimental Results

A sweep of three open-weight agents (DeepSeek-V3.1, Qwen3-235B-A22B-Instruct, Qwen2.5-72B-Instruct) was performed on the 40-task oracle-validated suite. Major findings:

Reliability collapse: DeepSeek-V3.1 achieves $\mathrm{pass}^1 = 0.67$, but only $\mathrm{pass}^5 = 0.38$—a 43% drop when requiring consistency across five runs. Both Qwen agents show smaller collapses, establishing that single-attempt accuracy overstates real-world, deployable competence.

Domain heterogeneity: Notable reversals and gaps emerged. DeepSeek is markedly superior in Project Accounting, while both Qwen models are dominant in Billing. All models fail at cross-system synchronization and AP coding/holds (hard frontier).

Model size does not guarantee performance: Qwen2.5-72B numerically outperforms the larger Qwen3-235B on both $\mathrm{pass}^1$ and $\mathrm{pass}^5$, although confidence intervals overlap.

Money-movement guard compliance: All scored models comply, no forbidden movement executed.

Project Management surface diversification: Addition of four PM apps (RedTeam, ingenious.build, Ressio, SmarteBuild) preserved model ranking and surfaced platform-specific task difficulty, especially in multi-app cross-system reconciliation.

Cost efficiency: Open-weight sweeps cost $\mathrm{pass}^1$0\$\mathrm{pass}^1$1\sim$\mathrm{pass}^1$212–\$57 sweep for frontier models).

---

Implications and Future Directions

Practical Implications

CFAgentBench enables rigorous measurement of finance agent reliability—not demonstration—across realistic, reconciled software stacks. The explicit money-movement guard directly reflects best practices in internal controls. The observed reliability collapse is a cautionary signal: deploying agents based on single-attempt leaderboards is misleading. Controllers should demand $\mathrm{pass}^1$3 reliability, not capability alone.

Theoretical Perspectives

This benchmark grounds the evaluation of vertical-domain agents in functional correctness and safety constraints, providing a model-neutral testbed for architectural improvements, tool-use routines, and controls-integrated policies. Reliability collapse and domain heterogeneity signal deep misalignment between surface-level text plausibility and robust, composable action planning—a challenge for agent architectures in high-consequence workflows. The money-movement guard is an operationalization of system-level safety protocols, not merely dataset annotation.

Future Directions

• Scaling up the executable suite to the full public split ($\mathrm{pass}^1$4) for high-powered model ranking.
• Introducing proprietary frontier models (Claude, GPT-4 family) with open-weight comparison to test generalization of reliability collapse.
• High-fidelity Tier-B/C deployments for more realistic system simulation.
• Gold trajectory demonstration and detailed failure analysis (defect clustering, money-guard behavior splits).
• Quarterly refresh of contamination-protected splits, guarding against train-on-test leakage.

---

Conclusion

CFAgentBench establishes a reproducible, functionally-graded, controls-driven benchmark for autonomous construction-finance agents. Its principled environment, task groundedness, and layered evaluation surface robust signals: single-attempt accuracy overstates competence, reliability collapses under repetition, and domain/task heterogeneity exposes architectural weaknesses. The money-movement guard is central—safety by default is measured as a first-class outcome, not a mere aspiration. Future research must extend reliability metrics and contamination controls, intersect agent architectures with system-level safety modeling, and shift the field toward deployable competence under real-world finance constraints. CFAgentBench will serve as a living benchmark, supporting open, rigorous comparison across models and promoting standards of reliability and safety aligned with professional finance practice.