Eval-Actions Benchmark: Robotic Manipulation Evaluation

• Eval-Actions Benchmark is a comprehensive framework that evaluates robotic manipulation models using multi-modal data, including vision-action and vision-language-action modalities.
• It integrates rich trajectory annotations with human expert supervision and automated methods to assess fine-grained performance dimensions such as success, smoothness, safety, and efficiency.
• The evaluation leverages advanced metrics like Spearman’s Rank Correlation and automated semantic assessment (AutoEval) with chain-of-thought reasoning to ensure reproducible and actionable insights.

The Eval-Actions Benchmark is a rigorously constructed framework for trustworthy evaluation of robotic manipulation models, addressing both fine-grained execution quality and the authenticity of demonstrated behaviors. By integrating rich trajectory data, multi-faceted supervision signals, and new automated evaluation methods, Eval-Actions establishes a reproducible, high-resolution standard for the semantic verification of vision-action and vision-language-action policies (Liu et al., 26 Jan 2026).

1. Dataset Design and Construction

Eval-Actions features comprehensive data modalities and scenario coverage. The dataset comprises:

• Vision-Action (VA): Wrist-mounted and third-person RGB-D sequence data for each manipulation episode.
• Vision-Language-Action (VLA): VA modalities augmented with textual task descriptions.
• Human teleoperation and policy execution: Data collected from 20 human operators and multiple policy-controlled agents to document both authentic and synthetic behavior.

The dataset explicitly incorporates both success and failure cases, including terminal failures (e.g., object drops) and safety violations (e.g., collisions). Each trajectory is annotated with a binary “Success” flag and detailed via radar charts along four axes: Success, Smoothness, Safety, and Efficiency. The corpus covers 13,000 episodes (≈52 hours), with 62.6% successes and 37.4% failures, spanning over 150 distinct manipulation scenarios (both single-arm and bimanual). The Eval-Actions Small (EAS) subset provides multi-view kinematic data (wrist, head, third-person, 7/14-DoF) for balanced source discrimination tasks.

2. Supervision Protocols for Reliable Scoring

Three principal forms of human supervision underpin the benchmark's ground-truth:

• Expert Grading (EG): Ten robotics experts assign tiered ratings (Excellent/Good/Poor) based on the four core dimensions. Aggregate score: $S_e = (1/10) \sum_{i=1}^{10} G_i$, where $G_i$ is the expert $i$'s numeric score.
• Rank-Guided Preferences (RG): Experts provide batch-wise relative rankings of small video groups. The calculated score incorporates penalties for collisions and failures:

$$S_{\text{raw}}(\theta) = \frac{\sum_i w_i\,s'_i}{\sum_i w_i}, \ s'_i = \begin{cases} s_i/\lambda_i & \text{if violation (collision/failure)} \ s_i & \text{otherwise} \end{cases}$$

Distribution alignment to ground-truth is achieved via Z-score normalization:

$$S_{\text{final}} = \mu_{\text{human}} + \sigma_{\text{human}}\frac{S_{\text{raw}}(\theta^*)-\mu_{\text{raw}}}{\sigma_{\text{raw}}}$$

• Chain-of-Thought (CoT) Annotations: Experts produce free-form, domain-specific reasoning paragraphs culminating in structured outputs including numerical scores, success/failure, and source categories; for example,

<think> … reasoning … </think> Score: S, Success: O, Source: C

These serve both as answer rationales and as training data for AutoEval-Plus.

Kinematic correlates for smoothness, safety, and efficiency are computed via joint velocity and acceleration statistics, collision event counting, and episode timing.

3. Evaluation Metrics and Analytical Tasks

Eval-Actions employs robust metrics optimized for continuous, rank-based, and categorical outputs:

• Spearman’s Rank Correlation Coefficient (SRCC): Quantifies monotonic alignment between predicted and human expert/aggregate ranks:

$$\rho = \frac{\sum_{i=1}^N (r_{g,i}-\bar r_g)(r_{p,i}-\bar r_p)} {\sqrt{\sum_i(r_{g,i}-\bar r_g)^2}\sqrt{\sum_i(r_{p,i}-\bar r_p)^2}}$$

Used for both EG and RG protocols (monotonic agreement).

• Source Authenticity Classification: Binary classification (Policy vs. Teleoperation), built into AutoEval as an explicit head, evaluated via accuracy, F1-score, and AUC.
• Fine-grained Quality Radar Charts: Each trajectory is annotated for all four performance dimensions.

The design allows for multi-task, multi-modal evaluation—action quality, authentic behavior discrimination, and CoT interpretability.

4. AutoEval and AutoEval-Plus: Automated Semantic Assessment

The benchmark is paired with the AutoEval system, providing automated evaluation aligned with human expert judgments:

• Spatio-Temporal Aggregation: Intermediate keyframes between $f_i$ and $f_{i+1}$ are spatially composited into a $k \times k$ grid, then resized for encoder input, compactly encoding kinematic and spatial information.
• Kinematic Calibration Signal: Joint-space statistics are derived:
  • $v_t = q_t - q_{t-1}$ (velocity), $\alpha_t = v_t - v_{t-1}$ (acceleration)
  • $U_v = \max_j \mathrm{Var}_t(v_{t,j})$, $U_\alpha = \max_j \mathrm{Var}_t(\alpha_{t,j})$, $\mu_v = \frac{1}{TJ} \sum_{t,j} |v_{t,j}|$
  • These statistics are fused into the encoder as a physical input prompt $I_{\text{phys}}$.
• Chain-of-Thought Reasoning and Group Relative Policy Optimization (GRPO):
  • CoT policies $\pi_\theta$ generate multi-token rationales and final answer blocks.
  • Reward signals are defined over agreement with human grades, success/failure, and provenance, aggregated as:

  $$R_{\text{acc}} = \omega_{\text{score}} R_{\text{score}} + \omega_{\text{succ}} R_{\text{succ}} + \omega_{\text{src}} R_{\text{src}}$$ - The GRPO objective incorporates relative advantages within ranked video batches and a KL regularization term:

  $$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{x\sim\mathcal{D}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{\pi_\theta(y_i|x)}{\pi_{\text{old}}(y_i|x)}A_i - \beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}}) \right]$$

5. Benchmark Performance and Empirical Results

AutoEval and competing systems are evaluated using EG and RG supervision protocols:

Method	EG SRCC	RG SRCC
InternVL3.5-4B	0.80	0.81
QwenVL3-4B	0.78	0.82
AutoEval-S	0.81	0.84

• Source Discrimination Accuracy: AutoEval-S attains 99.1% (EG) and 99.6% (RG) accuracy for trajectory provenance labeling.
• Ablation Studies: The best aggregation performance was observed with 8-frame, $2\times2$ grid stitching (0.84 RG SRCC). Larger grid sizes and omission of key modules (e.g., aggregation, GRPO, vision modality) yielded significant performance drops (e.g., kinematics only: SRCC 0.54 versus vision: 0.81).
• Interpretability: CoT outputs allow direct parsing of reasoning steps underlying outcome scores.

6. Comparison to Related Benchmarks and Extensions

Eval-Actions is distinguished by (1) explicit combination of vision, language, kinematics, and expert supervision, (2) the inclusion of failures, and (3) its focus on trust—source authenticity and fine-grained performance beyond binary success/failure. In contrast to traditional datasets limited to successful demonstrations, and in contrast to reasoning benchmarks like ActionReasoningBench—focused on logical state tracking, executability, ramifications, and action effects in simulated tasks (Handa et al., 2024)—Eval-Actions targets real-world execution evaluation under the multidimensional criteria that determine trustworthiness.

A plausible implication is that Eval-Actions, when paired with automated semantic assessment (AutoEval, AutoEval-Plus), provides a reproducible, scalable standard for model selection, benchmarking, and progress tracking in robotic manipulation, with extensibility to new actions, sensors, and supervisory regimes.

7. Prospects and Limitations

Examination of the Eval-Actions protocol reveals high sample diversity, robust failure structure, and tightly coupled evaluation infrastructure. Limitations, not explicitly addressed in the core framework, may include domain specificity (robotic manipulation focus), dependence on human-expert annotation for ground-truth, and fixed coverage of task scenarios and robot configurations. Continuous expansion to multi-agent, non-deterministic, and cross-domain manipulation tasks, as well as refinements in automated free-form answer evaluation, represent likely directions for further extension (Liu et al., 26 Jan 2026).