SeeAct Baselines in Autonomous Testing

• SeeAct Baselines are autonomous testing agents that use a monolithic, prompt-driven LLM invocation to execute and verify web test scripts.
• The architecture integrates test case initialization, dynamic observation via screenshots, and natural language prompts to control action execution in web applications.
• Empirical evaluations highlight limitations in assertion verification and step sequencing, prompting comparisons with more modular architectures like PinATA.

SeeAct baselines refer to a class of autonomous end-to-end testing agents designed by minimally adapting the SeeAct Autonomous Web Agent framework for the purpose of benchmarking LLM-powered Autonomous Test Agents (ATAs). The SeeAct-ATA baseline, as introduced in “Are Autonomous Web Agents Good Testers?” (Chevrot et al., 2 Apr 2025), represents a monolithic, prompt-centric approach for web application test script execution and serves as a point of comparison for subsequent, more modular ATA architectures.

1. Architectural Foundation of SeeAct-ATA

SeeAct-ATA originates from SeeAct’s single-prompt design, collapsing the classical AWA modules—profile, memory, planning, and action—into a single prompt-driven LLM invocation. The execution workflow comprises several stages:

• Initialization: The agent loads a natural-language test case (comprising a sequence of ordered steps with optional explicit assertions) and resets the application browser to the home page.
• Observation: After each agent action, a screenshot and optionally the DOM are captured for perceptual context in subsequent LLM prompts.
• Prompting: The LLM receives a prompt embedding the entire test case, a running history of previous actions, the most recent screenshot, and explicit, structured instructions. These instructions enforce step tracking, assertion enumeration and verification ({VERIFIED, NOT VERIFIED}), and request the next action (including element selection via multiple-choice, action type among {CLICK, TYPE, SELECT, PRESS ENTER, TERMINATE, NONE}, and optional value input).
• Execution: Actions proposed by the LLM are executed via a Playwright-based interaction engine that attempts to ground the referenced element literal in the DOM, then the success or failure of each is recorded.
• Loop: The cycle repeats until the LLM outputs TERMINATE or selects NONE, or cannot locate a required DOM element.

All scenario logic, including next-action selection, assertion control, and test step progress tracking, is mastered by the LLM within the prompt. There is no separation between planning, grounding, or assertion modules.

2. Decision Process and Control Flow

The control logic for SeeAct-ATA can be abstracted by the following high-level pseudocode (as stated in (Chevrot et al., 2 Apr 2025)):

load(TEST_CASE) previous_actions ← [] while True: screenshot ← capture_screenshot() prompt ← build_prompt(TEST_CASE, previous_actions, screenshot) response ← LLM(prompt) (element, action, value) ← parse(response) if action == TERMINATE or element == NONE: verdict ← PASS if all assertions VERIFIED else FAIL break success ← execute_action(element, action, value) previous_actions.append((element, action, value, success)) if current_step_assertion NOT VERIFIED: verdict ← FAIL; break return verdict

Logical constraints such as enforcing ordered steps and verifying intermediate outcomes are governed entirely through LLM prompt instructions, without recourse to explicit external planners or assertion modules.

3. Benchmark Composition and Evaluation Criteria

For empirical assessment of SeeAct-ATA, a comprehensive benchmark was constructed consisting of three offline web applications, each containerized via Docker to ensure reproducibility:

Application	Domain	Passing Tests	Failing Tests
Classifieds	Marketplace (~65k items)	15	15
Postmill	Social forum	18	16
OneStopShop	E-commerce	29	20

A total of 62 passing test cases were generated by four ISTQB-certified testers using a two-column step/assertion format. Failing cases (n=51) were created by single-point perturbations in passing scripts (e.g., targeting a non-existent element), yielding 113 test scenarios in total. Success requires strict in-order action execution and correct, complete assertion verification; omission or mis-verification constitutes failure.

4. Quantitative Performance Metrics

Test outcomes were compared to human verdicts and classified as true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). The principal metrics are:

• $$\mathrm{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$
• $\mathrm{Specificity} = \frac{TN}{TN + FP}$
• $\mathrm{Sensitivity} = \frac{TP}{TP + FN}$
• Step Mismatch Error Rate: $\mathrm{SMER}$
• True Accuracy: $\mathrm{TrueAcc}$

SeeAct-ATA (GPT-4o, temperature=0) achieves the following averaged results:

Metric	Classifieds	Postmill	OneStopShop	Average
Accuracy	0.50	0.56	0.59	0.55
Specificity	0.33	0.67	0.76	0.59
Sensitivity	0.67	0.44	0.35	0.48
SMER	0.45	0.20	0.17	0.28
TrueAcc	0.20	0.47	0.53	0.40

A plausible implication is that monolithic LLM approaches, while functional for simple scenarios, have significant reliability limitations in assertion validation and step order enforcement.

5. Comparative Analysis: SeeAct-ATA vs. PinATA

PinATA, a more advanced modular ATA, decomposes agent responsibilities into orchestrator, actor, and assertor components:

• Planning with feedback enables explicit step ordering and the possibility of action retries.
• Set-Of-Marks grounding and direct DOM/visual inspection provide richer action-to-element mapping.
• Dedicated assertion module re-invokes the LLM as an independent judge over screenshots.

PinATA's metrics (also on GPT-4o) reflect substantial improvements:

• Accuracy = 0.71 (+29 percentage points)
• Sensitivity = 0.88 (+40 points)
• SMER = 0.11 (–60%)
• TrueAcc = 0.61 (+53%)

On Postmill, PinATA achieves 94% sensitivity and approximately 60% overall true accuracy—a relative improvement of nearly 50% compared to SeeAct-ATA.

6. Fundamental Limitations and Recommendations for Robustness

SeeAct-ATA’s monolithic prompt architecture is characterized by:

• Single-prompt reasoning leads to frequent hallucination (HER ≈ 0.07), and erroneous step execution (SMER ≈ 0.28).
• No explicit grounding of action elements, increasing susceptibility to DOM mis-alignment errors.
• Passive visual assertion control resulting in mis-verification, particularly where content is dynamic or subtly altered.
• Lack of recovery mechanisms: Actions are executed in a single shot, precluding retry or rollback in the presence of execution failures.

To address these deficiencies and approach robust, low-maintenance test automation, recommended architectural enhancements include decomposition of planning/actuation/assertion roles, enhancement of DOM grounding, addition of assertion modules with action capabilities (e.g., scrolling, hover, direct DOM querying), and the adoption of structured scenario memory to enforce strict step-wise conformity and reduce premature side-effects (Chevrot et al., 2 Apr 2025). This suggests that future research should prioritize modular agent designs and rich perception-grounded action mechanisms for more reliable LLM-powered autonomous test agents.