GUIWorld: Dynamic GUI Modeling

• GUIWorld is a research domain modeling dynamic, multimodal graphical interfaces using large-scale, annotated video benchmarks for comprehensive state-action prediction.
• The annotation pipeline combines human review with MLLM-generated drafts to ensure high-fidelity QA and bounding-box localization, achieving strong inter-annotator agreement.
• Code2World advances GUI simulation by generating renderable HTML/CSS code to accurately predict next-step transitions with high action consistency and structural fidelity.

Graphical User Interface World (GUIWorld or GUI NC) refers to the emerging research area dedicated to modeling, understanding, and simulating graphical user interfaces as dynamic, multimodal environments for intelligent agents. This domain encompasses large-scale, annotated video benchmarks, methods for multimodal state-action-state prediction, and new architectures designed for temporally and structurally faithful GUI reasoning. Central artifacts include the GUI-World dataset (Chen et al., 2024), capturing diverse real-world applications, and Code2World, a code-generation-based world model for predicting GUI transitions (Zheng et al., 10 Feb 2026).

1. Dataset Design and Composition

GUI-World is a multimodal video benchmark targeting the comprehensive evaluation and development of agents that perceive and manipulate GUIs across static and dynamic settings. The dataset comprises:

• Six GUI Scenarios: Android, iOS, Desktop Software, Websites, Multi-Window workflows, and immersive XR headsets (such as Apple Vision Pro).
• Eight Question Types: Detailed Captioning, Summarized Captioning, Static-Content Q&A, Dynamic-Content Retrieval, Prediction (next-step forecasting), Sequential Reasoning, Multi-Round Conversation, Multiple-Choice Reasoning.
• Three Annotation Formats: Free-form Q&A, Multiple-Choice QA, Bounding-Box Localization.

Key quantitative statistics:

$$\begin{aligned} &\text{Total videos: } N = 12{,}379 \ &\text{Total keyframes: } 83{,}176 \implies \bar f \approx 6.72 \text{ frames/video} \ &\text{Total QA pairs: } 76{,}673 \text{ (free-form)} + 24{,}758 \text{ (MCQA)} + 24{,}758 \text{ (conversation)} = 126{,}189 \end{aligned}$$

Videos are 5–10 seconds long to capture atomic GUI workflows, with a nominal frame rate of 24 fps and input frames resized to $224 \times 224$ for model training (original up to $1920 \times 1080$). Dataset split is 80/10/10 (train/val/test).

GUI-World constitutes the first unified benchmark supporting rigorous study of dynamic, sequence-based, and multimodal GUI interpretation at scale.

2. Annotation Pipeline and Quality Assurance

Annotation follows a human–MLLM collaborative protocol:

1. Video and Keyframe Preparation: Student workers segment workflows and extract keyframes that mark atomic GUI operations.
2. Manual Annotation: Each frame is labeled with mouse actions (click/drag/hover/scroll), keyboard operation, and a natural-language sub-action rationale.
3. MLLM Generation: GPT-4V generates draft captions and Q&A content based on annotated keyframe sequences.
4. Human Review: Annotators correct LLM outputs, control for hallucinations, and refine question diversity and answer quality, applying a subsequent refinement prompt.

Quality metrics:

• Inter-annotator agreement: Measured on 1,000 video samples, $\kappa \approx 0.85$ (strong consistency) via Cohen’s kappa:

$\kappa = \frac{P_o - P_e}{1 - P_e}$

• Human verification: 98% of GPT-4V–generated captions accepted without substantive revision.
• Bounding-box QA: On a 100-video subset, [email protected] ≈ 0.92 for localization tasks.

This pipeline demonstrates the feasibility of large-scale, high-fidelity multimodal annotation involving both automated drafting and expert post-processing.

3. Benchmarking and Quantitative Evaluation

GUI-World enables systematic evaluation of state-of-the-art ImageLLMs and VideoLLMs. Key systems:

• ImageLLMs: GPT-4V(ision), GPT-4o, Gemini-Pro-1.5, Qwen-VL-Max (evaluated at the keyframe level).
• VideoLLMs: ChatUnivi, MiniGPT4-Video, VideoChat2 (consume raw video input).

Metrics:

• MCQA: Standard accuracy.
• Free-form and Conversational QA: LLM-as-judge score $s \in [1,5]$; BLEU and BERTScore for text matches.
• Bounding-box Tasks: Precision, Recall, F1.

Selected results (across all GUI-World scenarios):

Model	MCQA Acc. (%)	Free-form Score
GPT-4o	84.8	3.573
GPT-4V	83.5	3.543
Gemini-Pro-1.5	81.7	3.223
Qwen-VL-Max	74.0	2.553
VideoChat2	42.9	2.147
ChatUnivi	22.4	2.317
MiniGPT4-Video	16.8	1.443

Commercial ImageLLMs (>80% MCQA) outperform open-source VideoLLMs (<45% MCQA), highlighting a substantial domain gap and the challenges of modeling temporal GUI content (Chen et al., 2024).

4. World Modeling via Renderable Code: Code2World

Code2World (Zheng et al., 10 Feb 2026) advances next-step GUI prediction by reframing the task as renderable code generation rather than direct pixel or symbolic forecasting. The world model factors the transition as:

$C = M_\theta(s,a),\quad s' = R(C)$

where $s$ is the screenshot, $a$ is the user action (graphically marked), $C$ is HTML-like code, and $224 \times 224$0 is an HTML renderer outputting the next GUI image.

Architecture Overview:

• Backbone: Qwen3-VL-8B encodes $224 \times 224$1, where $224 \times 224$2 is an optional task goal.
• Decoder: Generates HTML/CSS for the next GUI state.
• Training:
  • Stage 1: Supervised Fine-Tuning (SFT) on $224 \times 224$3 pairs.
  • Stage 2: Render-Aware Reinforcement Learning (RARL) optimizes rewards based on:
  • Visual-Semantic Fidelity: Similarity of $224 \times 224$4 to $224 \times 224$5 (VLM-based judge, e.g., SigLIP score).
  • Action Consistency: Whether the transition preserves the user action ($224 \times 224$6), verified by an inverse-dynamics VLM.

The AndroidCode dataset supporting Code2World comprises 80,324 high-fidelity $224 \times 224$7 triplets. Initial HTML code is synthesized from screenshots (GPT-5), iteratively refined using visual-feedback loops until the rendering matches ground truth (threshold SigLIP ≥ 0.9).

Evaluation Metrics:

• Action Adherence (Sad): Correct action transition rate.
• Action Identifiability (Sid): Inference accuracy of $224 \times 224$8 from the predicted state.
• Element Alignment (Sele): UI element positioning accuracy (VLM-judged).
• Layout Integrity (Stay): Overall DOM structure fidelity.

Results show Code2World-8B achieves Sad = 94.28%, Sid = 88.64%, Sele = 71.35%, Stay = 70.32% (AndroidControl in-domain), surpassing all open-source VLMs and matching leading closed-source models. On out-of-distribution benchmarks (GUI Odyssey), only minor performance degradation occurs.

Plugging Code2World into downstream navigation tasks produces +9.5% success rate improvement for Gemini-2.5-Flash (41.4% → 50.9%) on AndroidWorld, and +2–5% for other agents, demonstrating plug-and-play benefit.

5. Model Architectures and Training Paradigms

GUI-Vid (GUI-World’s video LLM baseline) consists of:

• Vision Encoder: UMT-L extracting per-frame features.
• QFormer: 96 query tokens for instruction-guided context summarization.
• LLM Backbone: Vicuna with LoRA adapters; joint fine-tuning optimizes:

$224 \times 224$9

Hyperparameters include AdamW (learning rate 2e-5, batch 4, 3 epochs), eight input keyframes at $1920 \times 1080$0, and LoRA rank 8.

GUI-Vid provides +30% MCQA accuracy relative to VideoChat2 (→ 55% vs. 43%) and +0.65 free-form score improvement. Despite gains, it remains ≈25% behind GPT-4V and Gemini in MCQA.

Failure Modes:

• Misses fleeting GUI elements (pop-ups, progress bars).
• Difficulty maintaining temporal alignment and multi-window context.
• Small cursor motions (click vs. drag) are conflated.

Code2World employs a vision-language transformer for code generation, leveraging RL for human-like, action-consistent prediction.

Ablation studies indicate that combining visual-semantic and action-consistency rewards yields optimal fidelity and logic. Remaining challenges include scaling to native GUI frameworks and reducing judge-model cost.

6. Open Challenges and Prospective Directions

GUI-World and Code2World expose fundamental challenges in dynamic GUI reasoning:

• Static vs. Dynamic Understanding: ImageLLMs excel at static keyframe tasks; dynamic and sequential reasoning (dynamic-content QA, next-step prediction) remain unsolved for both open and closed-source VideoLLMs (Chen et al., 2024).
• Data Modality Gap: VideoLLMs pretrained on natural videos underperform on structured GUI data, underscoring the need for synthetic, operation-rich pretraining streams.
• Temporal Modeling: Accurate keyframe extraction and precise temporal context remain critical bottlenecks—especially as GUIs exhibit small, high-salience changes.
• Auxiliary Modalities: Integration of OCR and operation-history logs with vision models is proposed to enhance cross-modal consistency.
• Planning Integration: Tight coupling of world-models (i.e., code-based simulators) with agent planners could facilitate robust, foresightful navigation in complex apps (Zheng et al., 10 Feb 2026).

A plausible implication is that future advances will require architectures blending action-conditioned code synthesis, multimodal fusion (vision, text, history logs), and RL-guided reward shaping for both logic and fidelity.

7. Significance and Resources

GUI-World provides a unified, scalable platform for benchmarking dynamic, multimodal GUI understanding, with broad coverage across device classes, task types, and real-world scenarios. Code2World demonstrates that renderable code generation enables both structural and visual precision in UI state emulation, setting a new standard for action-conditioned GUI simulation.

Datasets and codebases:

• GUI-World: https://gui-world.github.io/ (Chen et al., 2024)
• Code2World: https://github.com/AMAP-ML/Code2World (Zheng et al., 10 Feb 2026)

These resources lay the empirical foundation for advancing intelligent agents capable of robust, generalizable GUI interaction and planning in both research and industrial settings.