GENIUS: Generative Fluid Intelligence Evaluation Suite

Abstract: Unified Multimodal Models (UMMs) have shown remarkable progress in visual generation. Yet, existing benchmarks predominantly assess $\textit{Crystallized Intelligence}$, which relies on recalling accumulated knowledge and learned schemas. This focus overlooks $\textit{Generative Fluid Intelligence (GFI)}$: the capacity to induce patterns, reason through constraints, and adapt to novel scenarios on the fly. To rigorously assess this capability, we introduce $\textbf{GENIUS}$ ($\textbf{GEN}$ Fluid $\textbf{I}$ntelligence Eval$\textbf{U}$ation $\textbf{S}$uite). We formalize $\textit{GFI}$ as a synthesis of three primitives. These include $\textit{Inducing Implicit Patterns}$ (e.g., inferring personalized visual preferences), $\textit{Executing Ad-hoc Constraints}$ (e.g., visualizing abstract metaphors), and $\textit{Adapting to Contextual Knowledge}$ (e.g., simulating counter-intuitive physics). Collectively, these primitives challenge models to solve problems grounded entirely in the immediate context. Our systematic evaluation of 12 representative models reveals significant performance deficits in these tasks. Crucially, our diagnostic analysis disentangles these failure modes. It demonstrates that deficits stem from limited context comprehension rather than insufficient intrinsic generative capability. To bridge this gap, we propose a training-free attention intervention strategy. Ultimately, $\textbf{GENIUS}$ establishes a rigorous standard for $\textit{GFI}$, guiding the field beyond knowledge utilization toward dynamic, general-purpose reasoning. Our dataset and code will be released at: $\href{https://github.com/arctanxarc/GENIUS}{https://github.com/arctanxarc/GENIUS}$.

• The paper formalizes generative fluid intelligence using three primitives: implicit pattern induction, ad-hoc constraint execution, and contextual knowledge adaptation.
• It introduces a benchmark of 510 multimodal, expert-curated samples with a hybrid evaluation protocol focusing on rule compliance, visual consistency, and aesthetic quality.
• Empirical evaluations show state-of-the-art UMMs underperform, revealing an execution gap that calls for advanced attention modulation and context-sensitive model innovations.

GENIUS: A Rigorous Benchmark for Generative Fluid Intelligence in Unified Multimodal Models

Formalization and Distinction of Generative Fluid Intelligence

The paper "GENIUS: Generative Fluid Intelligence Evaluation Suite" (2602.11144) addresses a foundational oversight in the evaluation of Unified Multimodal Models (UMMs): while recent breakthroughs have enabled highly competent visual generation via the fusion of language and vision modalities, current benchmarks almost exclusively assess Crystallized Intelligence (CI)—memorization and retrieval based on extensive pre-training. This focus neglects the dimension most closely linked with general intelligence: Fluid Intelligence (FI), the ability to induce, reason, and dynamically adapt in unfamiliar scenarios.

GENIUS formally introduces and operationalizes Generative Fluid Intelligence (GFI) within visual generation. Drawing from the Cattell-Horn-Carroll (CHC) theory, GFI is synthesized from three primitives:

• Implicit Pattern Induction: Inferring latent, personal or abstract preferences from interleaved context and reproducing them in generated outputs.
• Ad-hoc Constraint Execution: Dynamic reasoning and application of novel, context-specific symbolic or visual rules, decoupled from pre-trained semantics.
• Contextual Knowledge Adaptation: Flexible generation based on newly defined knowledge or counterfactual instructions, necessitating inhibition of intrinsic priors.

This rigorous definition fills an important theoretical gap and provides conceptual clarity needed for steering future advancements.

GENIUS Benchmark Construction

GENIUS comprises 510 multimodal, expert-curated samples spanning three dimensions and five tasks, systematically isolating GFI from CI. Each task couples multi-image and interleaved textual context, creating highly coupled, information-dense scenarios in which ad-hoc definitions, rules, and preferences are provided on-the-fly. Importantly, the benchmark is meticulously designed to strictly decouple prior knowledge: each instance cannot be solved by recalling pre-trained information or static concept schemas.

The evaluation protocol is hybrid: three orthogonal metrics are assessed using a Large Multimodal Model (LMM) as judge, cross-validated with human annotation. Metrics include:

• Rule Compliance (RC): Strict adherence to ad-hoc rules and constraints.
• Visual Consistency (VC): Preservation of contextual visual elements under transformation.
• Aesthetic Quality (AQ): Maintenance of anatomical logic and visual realism.

Scores are aggregated with a weighted ratio (RC:VC:AQ = 6:3.5:0.5), reflecting the primacy of logical and context-grounded compliance in GFI.

Empirical Evaluation and Diagnostic Analysis

GENIUS provides a robust, fine-grained assessment of twelve representative UMMs, including both proprietary (Nano Banana Pro, GPT-Image, SeeDream) and open-source (Bagel, Qwen-Image, GLM-Image, FLUX.2-dev) architectures. The quantitative results reveal a striking deficiency: even the strongest proprietary models (Nano Banana Pro) fall below a passing grade (overall score: 57.19), while open-source exemplars (Bagel) are significantly weaker (overall score: 26.74).

Notably, performance in Contextual Knowledge Adaptation tasks is consistently lower, reflecting an inability to override pre-trained priors when faced with context-defined rules or counterfactual knowledge. Empirical ablations further demonstrate that inference-time strategies like pre-planning or post-reflection yield only marginal improvements.

A key insight emerges from diagnostic probes: models exhibit a pronounced "illusion of competence"—high scores on Aesthetic Quality mask severe deficits in Rule Compliance and context comprehension. This suggests optimization has disproportionately targeted surface-level visual plausibility, failing to equip models with general-purpose reasoning or dynamic adaptation.

The paper further demonstrates, through VQA reformulation, that generative failure stems primarily from execution gaps rather than comprehension deficits: models often understand the task intent but cannot faithfully synthesize compliant visual outputs, especially under information-dense, interleaved contexts.

Theoretical Analysis and Attention-Based Remedy

The root cause of GFI failure is traced to an imbalanced and noisy attention distribution across multimodal context tokens. Using theoretical constructs from In-Context Learning (ICL) as Implicit Fine-Tuning, the authors formalize the equivalence between context attention and implicit gradient updates. They prove (see Theorem 4.1 and 4.2) that the magnitude and direction of attention directly modulate the gradient norms during generation, dictating whether adaptation tracks the context signal or default priors.

Leveraging this, a training-free attention intervention is proposed, comprising three stages: Keyword Distillation (to identify signal tokens), Relevance Mapping (to quantify contribution), and Bias Injection (to dampen noise tokens in the attention logits). This tactic efficiently suppresses irrelevant gradient components, steering the generative trajectory toward rule-compliant adaptation without retraining.

Experimental results validate performance gains (+6.18% overall score for Bagel baseline), with attention visualization confirming a sharpened focus on critical context regions. This demonstrates the practical feasibility of boosting GFI within existing architectures.

Implications and Future Directions

GENIUS establishes a rigorous standard for GFI in visual generation, exposing a fundamental gap in current UMM architectures. The findings carry several implications:

• Practical: GENIUS can serve as a licensed diagnostic testbed for model selection, architecture design, and training paradigm evaluation where general intelligence, adaptability, and context-driven reasoning are essential (creative assistants, scientific visualization, high-stakes AI deployment).
• Theoretical: The benchmark enables a clear delineation between CI and FI in generative domains, opening avenues for further research on implicit context processing, dynamic inhibition of priors, and architecture-level strategies for meta-learning.
• Societal: By highlighting the illusion of competence, GENIUS prevents premature deployment of fragile systems in settings requiring rule-bound control and transparent evaluation.

A central challenge remains: the execution gap between context comprehension and generative adaption. Future development of attention modulation, gradient-aware optimization, and context-sensitive architectural innovations will be critical for advancing fluid intelligence in generative AI.

Conclusion

GENIUS delivers a formal definition, systematic benchmark, and practical remedial strategy for Generative Fluid Intelligence in UMMs. It exposes deep-seated limitations in current models and establishes an actionable roadmap for bridging the gap between crystallized knowledge retrieval and true general intelligence in visual generation. The benchmark is well-positioned to influence both theoretical inquiry and practical advances in adaptive, logic-grounded AI systems.