WebJudge Protocol

• WebJudge Protocol is a framework that evaluates the reliability of LLMs and MLLMs in judging web development artifacts.
• It formalizes static and dynamic assessment methods through query-specific rubric trees and human-annotated pairwise preferences.
• The protocol benchmarks automated judging by diagnosing bias, functional equivalence errors, and leveraging live interaction for nuanced evaluation.

The WebJudge protocol defines a systematic framework for evaluating the reliability and discriminative power of LLMs, multimodal LLMs (MLLMs), and agentic workflows when used as judges of web development artifacts. It formalizes both static (non-interactive) and dynamic (interactive) paradigms for assessment, grounded in query-specific rubric trees and anchored by human-annotated pairwise preference data. The protocol aims to rigorously measure agreement between automated judges and human experts, diagnose model failure modes such as bias and functional equivocation, and provide reproducible procedures for benchmarking automated web task evaluation (Li et al., 21 Oct 2025).

1. Formal Problem Definition and Objectives

WebJudge models the meta-evaluation problem as the analysis of a dataset:

$$D = \{ (Q_1, W_1^a, W_1^b, \ell_1), \ldots, (Q_N, W_N^a, W_N^b, \ell_N) \}$$

where for each instance: $Q_i$ is a user-specified web development query, $W_i^a$ and $W_i^b$ are two independently generated web implementations responsive to $Q_i$, and $\ell_i \in \{\text{a}, \text{b}, \text{tie}\}$ captures the human-annotated preference. A Judge $J(\cdot)$ receives $(Q, W^a, W^b)$ and a rubric set $R$, outputting a predicted label $\hat{\ell} \in \{\text{a}, \text{b}, \text{tie}\}$ or scores $Q_i$0 for grading.

The main objectives are to:

• Measure empirical agreement: $Q_i$1
• Analyze error typologies (positional bias, functional equivalence misclassifications, feasibility errors)
• Contrast static and dynamic evaluation methodologies

2. Assessment Paradigms: Static and Dynamic

Static (Non-interactive) Protocol

The static paradigm operates on the fixed source code (and, optionally, a screenshot) for each implementation. Two canonical modes are used:

• Single-Answer Grading: For each $Q_i$2, the Judge computes a scalar score $Q_i$3. Pairwise preference is determined by thresholded differences:

$Q_i$4

• Pairwise Comparison: The Judge directly ingests $Q_i$5 and outputs $Q_i$6.

Pseudocode:

$Q_i$2

A Likert-variant allows for scoring across Functionality, UI, Code, and Interactivity, producing summed scalar scores.

Dynamic (Interactive) Protocol

In the interactive setup, the Judge functions as an agent operating within a live browser environment $Q_i$7 initialized with the candidate implementation $Q_i$8. The agent may issue actions (e.g., click, type, scroll, wait) and perceive the DOM tree and screenshot at each state $Q_i$9. Its workflow is decomposed into:

• Planner: Extracts testable tasks from the rubric.
• Executor: Conducts environmental interactions.
• Summarizer: Aggregates pass/fail task results.

Pseudocode:

$Q_i$3

Continuous operation allows adaptive task sequencing contingent on environment feedback.

3. Rubric Construction and Scoring Mechanisms

WebJudge employs query-grounded rubric trees $W_i^a$0 shaped around three axes:

1. Intention: Core end-user features.
2. Static Quality: UI components, layout, code structure.
3. Dynamic Behavior: Interactive task functionality.

Each non-leaf rubric subdivides into finer bits; leaves represent atomic, binary-checked criteria. For a leaf $W_i^a$1:

$W_i^a$2

where 1 if $W_i^a$3 implements $W_i^a$4 under $W_i^a$5, 0 otherwise.

Aggregates:

• Dimension score:

$W_i^a$6

• Overall rubric score (default uniform weights $W_i^a$7):

$W_i^a$8

Each rubric tree $W_i^a$9 is specialized to $W_i^b$0 and LLM-generated, then manually validated for consistency.

4. Data Resource and Human Preference Annotation

WebJudge's base dataset derives from "webdev-arena-preference-10k", curated through a rigorous two-phase filtering:

• Query Filtering: Removal of duplicates, unsafe/infeasible/ambiguous queries.
• Environment Filtering: Deploy in a controlled Next.js setup; exclude load failures or blank renders (via screenshot validation).

Annotations involve two expert raters, each conducting rubric-guided preference assignments over $W_i^b$1 tuples. Emphasis is placed on:

• Functionality prioritized over UI aesthetics
• Avoiding ties except in cases of strict equivalence
• Allowing for functionally equivalent, non-identical structural solutions

Inter-annotator raw agreement $W_i^b$2. Cohen’s kappa is calculated as:

$W_i^b$3

with $W_i^b$4 as chance-agreement.

5. Metrics and Analytical Approaches

The protocol prescribes several evaluative metrics:

• Agreement Accuracy:

$W_i^b$5

• Spearman’s $W_i^b$6 (scalar score ranking correlation):

$W_i^b$7

• Precision, Recall, F1 for specific rubric dimensions (e.g., feasibility):

$W_i^b$8

• Statistical significance assessments: bootstrap confidence intervals for Acc; McNemar’s test for judge comparison.

6. Execution in Dynamic Environments and Agentic Workflow Specification

The $W_i^b$9 environment is instantiated via a headless or GUI browser under standard Next.js runtime constraints. Judge agent actions comprise an LLM-to-action translation layer (such as ReAct paradigm) producing commands (click, type, scroll, wait) interpretable by control frameworks like pyautogui.

The agentic loop repeatedly observes $Q_i$0, consults $Q_i$1, plans and executes next tasks, then aggregates outcomes for final summarization. This enables real-time adaptation to environment feedback and more nuanced, context-sensitive evaluation.

7. Model Limitations and Error Taxonomy

Multiple systematic failure modes are identified:

• Positional Bias: Automated judges systematically overprefer the first or second implementation by approximately 5–10%, irrespective of checklist instructions.
• Functional Equivalence Failure: Judges frequently penalize solutions solely due to naming or presentation variations, despite identical underlying semantics (e.g., labeling “Presentation” versus “Demonstration”).
• Feasibility Verification Shortcomings: Static judges achieve high recall for feasibility at the cost of low precision, largely due to reliance on code patterns rather than live execution. Interactive agents invert this trend, demonstrating high precision but low recall, most often attributable to brittle navigation or rendering failures.
• Error Accumulation in Agentic Pipeline: Weaknesses in the plan–execute–summarize sequence, such as erroneous planning or execution, propagate, culminating in compounded evaluation errors.

Collectively, these constraints underscore substantive reliability gaps between current (M)LLM judges and human experts. The protocol thereby establishes comprehensive benchmarks, highlighting both methodological advances and the extant challenges for automated web task evaluation (Li et al., 21 Oct 2025).