Cosmos-Reason Backbone

• Cosmos-Reason Backbone is a modular framework that integrates ontology-guided reasoning with closed-loop symbolic regression to advance both physical common sense and cosmological model discovery.
• It employs a multimodal pipeline combining visual encoding, text tokenization, and a hybrid Transformer architecture to effectively process richly annotated scientific data.
• The system incorporates chain-of-thought training, reinforcement learning, and Bayesian inference to iteratively refine and validate models against benchmark datasets.

The Cosmos-Reason backbone refers to a modular computational and modeling paradigm characterized by ontology-guided reasoning in physical AI systems and closed-loop, data-driven symbolic regression in cosmological modeling. Instantiated in both the Cosmos-Reason1 models for Physical AI (NVIDIA et al., 18 Mar 2025) and the CosmoGen cosmological model generator (Castelão et al., 18 Sep 2025), the backbone synthesizes richly annotated data design, multimodal LLM architectures, and fully automated evolutionary pipelines for scientific model discovery and evaluation.

1. Ontology-Guided Reasoning and Data Construction

Central to the Cosmos-Reason1 backbone is the explicit use of ontologies for structuring both the data and the evaluation benchmarks. For physical common sense reasoning, a hierarchical ontology is defined with three top-level nodes—Space, Time, and Fundamental Physics—and sixteen leaf subcategories, including Space (Relationship, Plausibility, Affordance, Environment), Time (Actions, Order, Causality, Camera, Planning), and Fundamental Physics (Attributes, States, Object Permanence, Mechanics, Electromagnetism, Thermodynamics, Anti-Physics). These ontological labels do not appear as explicit graph components in the model architecture; rather, they function as metadata and prompt annotations during dataset construction and evaluation. Each exemplar in training and testing is tagged with its relevant subcategory to facilitate coverage and benchmarking.

For embodied reasoning, a two-dimensional ontology is specified. Dimension one (Agent Type) spans human/animal, robot arms, humanoid robots, and autonomous vehicles. Dimension two encodes Reasoning Capability: (1) processing complex sensory inputs, (2) predicting action effects, (3) respecting physical constraints (affordance), and (4) learning from interaction (presently deferred). The ontological schema is used uniformly to create prompts and hierarchies for all agent types, ensuring coverage across embodiments. The ontology is integrated via prompt template engineering and curated labeling, not via dedicated embedding layers or latent factorization.

2. Multimodal Architecture and Pipeline

The Cosmos-Reason1-8B and Cosmos-Reason1-56B models embody a multimodal pipeline comprising vision, text, and reasoning:

• Visual Encoder: Video frames (≤32, 448×448 px, ≤2 fps) are processed by a frozen InternViT-300M-V2.5 encoder, generating 1,024 patch tokens per frame. PixelShuffle performs 2×2 spatial downsampling, resulting in 256 tokens/frame.
• Image Handling: Images are split into 1–12 tiles plus a thumbnail, each tile fed through the same encoder pathway.
• Text Tokenization: Text prompts leverage the LLM’s standard token vocabulary.
• Projector MLP: Features from the vision encoder are projected into LLM-compatible embedding spaces using a two-layer MLP (output dim 4,096 for 8B, 8,192 for 56B).
• Decoder-only Backbone: A hybrid network (Mamba-MLP-Transformer) alternates between linear-time state-space Mamba modules and multi-head self-attention Transformer blocks. The 8B variant comprises 52 layers (problem-matched mixing of Mamba and Transformer components), while the 56B variant comprises 118 layers.
• Parallelism and Optimization: The 8B model employs TP=4; 56B uses TP=8, PP=2; both use Adam with β₁=0.9, β₂=0.95, and weight decay=0.1.

Ontologies are incorporated exclusively through dataset and prompt construction; no graph neural networks, latent attention masks, or ontology-projection matrices are present in the backbone.

3. Chain-of-Thought Reasoning and Training Paradigm

Chain-of-Thought (CoT) generation is integral to Cosmos-Reason1. Each SFT data point consists of a video prompt, a question, an explicit multi-step CoT trace (human or model-generated), and a final answer. The model learns by maximizing the causal cross-entropy of the complete trace plus answer, with loss $$L_{sft} = -\sum_{(\text{video, question, CoT, answer})} \log P(\text{CoT} \| \text{answer} \mid \text{video, question})$$ (“$\|$” denotes sequence concatenation).

During inference, five outputs per prompt are sampled (temperature=0.6, top-p=0.95), and average accuracy is reported. Consistency in multistep reasoning is maintained solely by dataset structure; no auxiliary coherence regularization is imposed.

4. Reinforcement Learning Enhancement and Objective Functions

Physical AI reinforcement learning (RL) augments SFT by using Generalized Reward-penalized Policy Optimization (GRPO). For each multiple-choice question (MCQ) item, nine candidate outputs are sampled. Rewards are assigned as $R(o_i) = 1$ if the answer string matches ground truth (verified by regex), $0$ otherwise. The advantage is formulated $$A_i = [R(o_i) – \text{mean}_j R(o_j)]/\text{std}_j(R(o_j))$$, and the RL objective seeks to maximize $$\sum A_i \log \pi(o_i|\cdot) – \beta \cdot \mathrm{KL}(\pi\,||\,\pi_{ref})$$, with $\beta=0.005$. Training uses a learning rate $4 \times 10^{-6}$, batch size $128$ MCQs $\times 9$ samples, for $500$ iterations.

5. Performance Metrics and Benchmarking

Benchmarks constructed via the ontologies reveal substantive performance improvements:

Benchmark	Cosmos-Reason1-8B SFT	Cosmos-Reason1-56B SFT	8B SFT+RL
Physical Common Sense (604 Q)	52.3% (+6.9)	60.2% (+2.0)	55.1% (+2.8)
Embodied Reasoning (1214 Q)	60.0% (+12.8)	63.7% (+10.2)	67.1% (+7.1)
Intuitive Physics	65.7% (+23.4)	–	68.7% (+3.0)

Improvements are recorded relative to the backbone (pretrained) model and leading baselines (e.g., OpenAI o1). RL yields a further accuracy lift. However, models remain challenged by specialized datasets such as “RoboFail,” highlighting opportunities for future augmentation.

6. Cosmos-Reason Backbone in Data-Driven Scientific Modeling

The CosmoGen “Cosmos-Reason” backbone extends the paradigm to symbolic scientific discovery. A closed-loop pipeline integrates evolutionary symbolic regression (tree-based GP via DEAP), cosmological solvers (CLASS), and Bayesian inference (MontePython):

1. Model Generation: Analytic ansätze $f(a;D)$ for dark-energy density are algorithmically proposed and represented as symbolic trees.
2. Physical Filtering: Candidates are prefiltered for mathematical and physical consistency.
3. Simulation and Evaluation: Survivors are inserted as CLASS plugins to compute background $H^2(a)$ and linear perturbation observables ($P(k,z)$, $C_\ell$). Short MCMC runs extract best-fit $\chi^2$ values, targeting mitigation of $H_0$ and $S_8$ tensions:

$$\chi^2_{\rm tot} = \chi^2_{\rm Planck} + \left( \frac{H_0 - H_0^{\rm SH0ES}}{\sigma_{H_0}} \right)^2 + \left( \frac{S_8 - S_8^{\rm KiDS}}{\sigma_{S_8}} \right)^2$$

and $\mathrm{fitness} = \chi^2_{\rm tot}$.

1. Genetic Evolution: Parent pools are selected (tournament, $(\mu+\lambda)$ strategy), offspring are generated by subtree crossover ($p_{\rm mate}=0.5$) and point mutation ($p_{\rm mut}=0.5$), and complexity is controlled via tree-size limits.
2. Bayesian Validation: Full posterior exploration employs MontePython (Metropolis-Hastings) or Polychord for final model validation with likelihood

$$\mathcal{L}(\theta) = \exp\left[-\chi^2(\theta)/2\right]$$

and standard cosmological priors.

The backbone features modular interfaces: the SR engine exposes “propose_model() → AST,” translators convert symbolic trees to CLASS code, and MontePython scripts orchestrate evaluation and fitness feedback.

7. Extensions and Implications

A plausible implication is that the Cosmos-Reason backbone generalizes to any scientific modeling pipeline wherein model hypotheses can be generated algorithmically, evaluated numerically, and optimized in closed-loop fashion using ontological or domain-guided benchmarks. In both physical AI and cosmology, the backbone functions as an organizing principle for data curation (via ontologies), multimodal inference, and iterative hypothesis refinement.

Extensions include adaptation to new observables (modifying likelihoods), broadening to coupled dark energy/scalar-tensor theories, and interchangeability of symbolic regression engines, contingent on matching API interfaces and evaluation protocols. The approach is fully automated and data-driven, supporting both analytic model discovery and high-fidelity empirical reasoning.

• Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning (NVIDIA et al., 18 Mar 2025)
• CosmoGen: a cosmological model generator (Castelão et al., 18 Sep 2025)