Cosmos-Reason1: Physical AI Framework

• Cosmos-Reason1 is a framework that uses dual ontologies to define physical common sense and embodied reasoning, systematically addressing Space, Time, and Physics.
• It employs a decoder-only multimodal LLM with chain-of-thought inference, integrating vision and text streams for detailed stepwise physical reasoning.
• Comprehensive benchmarks and tailored training protocols validate improved performance and robust adaptation across diverse physical AI tasks.

Cosmos-Reason1 refers to the suite of models, benchmarks, and formal methodologies introduced in "Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning" (NVIDIA et al., 18 Mar 2025), explicitly designed to endow large multimodal LLMs with both physical common sense and embodied reasoning capabilities. The framework introduces a dual-ontology system to systematically represent the knowledge requirements for machines operating in physical environments, develops novel multimodal architectures supporting chain-of-thought (CoT) inference, and pioneers rigorous benchmarks for evaluating physical reasoning aligned with these ontologies. Cosmos-Reason1 thus forms a directly testable blueprint for advancing the Physical AI paradigm, where models must not only perceive but reason about, and act within, the physical world.

1. Ontological Foundations: Common Sense and Embodiment

Cosmos-Reason1 is anchored in an explicit ontological framework tailored to physical reasoning. The ontologies are twofold:

• Hierarchical Ontology for Physical Common Sense: Physical knowledge is categorized into three top-level domains—Space, Time, and Physics—with each decomposed into subcategories. For example, "Space" includes Relationship, Plausibility, Affordance, and Environment; "Time" comprises Actions, Order, Causality, Camera, and Planning; "Physics" branches into Attributes, States, Permanence, Mechanics, Electromagnetism, Thermodynamics, and AntiPhysics. Formally, this ontology is a tree $O = (C, S)$ with three roots ($C$) and finer-grained $S_{c_i}$. This specification enables systematic coverage and benchmark construction for all relevant commonsense microdomains.
• Two-Dimensional Ontology for Embodied Reasoning: Reasoning skills (e.g., ProcessInputs, PredictEffects, RespectConstraints) are arrayed against agent embodiments (Human, Animal, RobotArm, Humanoid, AutonomousVehicle), creating a matrix $$E_{i,j} = (\text{capability } r_i, \text{embodiment } a_j)$$. This structure explicitly decouples agent-specific constraints from the reasoning operations required, allowing for cross-embodiment generalization and targeted assessment.

These ontologies are formalized, directly encoded into the benchmark query sets, and used to guide both data curation and evaluation strategy (NVIDIA et al., 18 Mar 2025).

2. Model Architecture: Multimodal Chain-of-Thought

Cosmos-Reason1 introduces two primary models—Cosmos-Reason1-8B and Cosmos-Reason1-56B—both built as decoder-only multimodal LLMs:

• Vision Encoder: InternViT-300M, a 300M-parameter model, processes video data, with outputs projected down to LLM embedding size via a 2-layer MLP projector.
• LLM Backbone: Hybrid Mamba-MLP-Transformer architectures with 52–118 layers depending on scale, interleaving Mamba (linear time SSM) and attention blocks. The total parameter count ranges from 8B to 56B.
• Fusion: Visual and textual inputs are concatenated at the embedding level: visual tokens from processed frames are flattened and combined with text tokens, then passed to the LLM, which emits a CoT reasoning trace culminating in an answer.

The model is explicitly engineered for chain-of-thought output, enabling stepwise reasoning over multimodal context ("> ... "), mirroring the physical reasoning process defined in the ontologies (NVIDIA et al., 18 Mar 2025).

3. Training Protocols: Supervised Fine-Tuning and RL

Model training proceeds in four tightly specified phases:

1. Vision Pre-training: 130M image/video-text pairs, encoder frozen, cross-entropy loss.
2. General SFT: 6M image-text + 2M video-text instances for broad visual QA and captioning; full model tuned end-to-end.
3. Physical AI SFT: 3.76M custom-curated samples (1.82M understanding, 1.94M reasoning), synthesized to maximize spread across ontology branches. Tasks cover both commonsense VQA and embodied, next-action reasoning (e.g., predicting robot or vehicle actions in context), with strong augmentation for spatial puzzles, temporal direction, and object permanence.
4. Physical AI RL: On-policy RL using Generalized Reward-Policy Optimization (GRPO), leveraging MCQ correctness and answer format as reward. The policy is regularized by a KL penalty to a supervised reference.

Losses are standard cross-entropy (for SFT) and advantage-weighted policy loss with KL regularization (GRPO) for RL, carefully selecting reward normalization and batch sizes matched to the data statistics (NVIDIA et al., 18 Mar 2025).

4. Benchmark Construction and Empirical Evaluation

Benchmarks are systematically designed to instantiate all ontology branches, forming a comprehensive suite for both physical common sense and embodied reasoning:

• Physical Common Sense Benchmark: 426 Internet videos, 604 questions (336 binary, 268 MCQ), sampling the ontology's 13% Space, 49% Time, 37% Physics splits. Evaluation metric is final-answer accuracy.
• Embodied Reasoning Benchmark: 610 MCQs across six splits (BridgeData V2, RoboVQA, RoboFail, AgiBot, HoloAssist, Autonomous Vehicle), covering next action, affordance, and task-completion queries.
• Intuitive Physics Benchmark: Specialized MCQs for arrow-of-time, spatial puzzles, and object permanence.

Model comparison tables demonstrate substantial performance gains for Cosmos-Reason1-8B (SFT and RL) over baseline LLMs (e.g., Qwen2.5-VL, OpenAI o1, Gemini 2.0 Flash), with observed delta increases of 5–13% absolute accuracy in all major subdomains.

Benchmark	Cosmos-Reason1-8B (SFT)	Cosmos-Reason1-8B (+RL)	Leading Non-Cosmos Model
Physical Common Sense	52.3%	57.3%	59.9% (OpenAI o1)
Embodied Reasoning	60.0%	67.1%	47.2% (Backbone 8B)
Intuitive Physics (avg.)	65.7%	68.7%	58.3% (GPT-4o)

Empirical results confirm that RL-finetuned models exhibit consistent CoT reasoning improvements and generalization across both previously observed and novel physical contexts (NVIDIA et al., 18 Mar 2025).

5. Representative Reasoning Traces and Qualitative Behavior

Cosmos-Reason1 models generate detailed, step-structured reasoning traces that explicitly track ontology-aligned subdomains. Example outputs include:

• Spatial Relation: Reasoning that double yellow lines (Time—RespectConstraints) preclude a lane change in an AV task.
• Object Affordance: Deducing that an absent tomato in a "make a salad" task (Space—Affordance, Environment) means only cucumber can be prepared next.
• Physics Sanity: Identifying anti-gravity violations (Physics—AntiPhysics) in time-reversed video.

These traces interleave checks on object relations, planning, and physical law violations, matching the systematization of the ontological hierarchy and demonstrating genuine adaptation across embodiment and physical scenario (NVIDIA et al., 18 Mar 2025).

6. Open Source Ecosystem and Reproducibility

All code and pre-trained models for Cosmos-Reason1 are distributed under the NVIDIA Open Model License, with full dataset, benchmark, and training pipeline release at https://github.com/nvidia-cosmos/cosmos-reason1. This facilitates reproducibility, benchmarking, and further extension by both academia and industry (NVIDIA et al., 18 Mar 2025).

---

Cosmos-Reason1 thus represents a methodologically rigorous, ontologically explicit, and empirically validated framework for physical common sense and embodied reasoning in multimodal AI systems, establishing new standards for both representational completeness and practical performance in Physical AI (NVIDIA et al., 18 Mar 2025).