Self-Improving AI Agents through Self-Play

Abstract: We extend the moduli-theoretic framework of psychometric batteries to the domain of dynamical systems. While previous work established the AAI capability score as a static functional on the space of agent representations, this paper formalizes the agent as a flow $ν_r$ parameterized by computational resource $r$, governed by a recursive Generator-Verifier-Updater (GVU) operator. We prove that this operator generates a vector field on the parameter manifold $Θ$, and we identify the coefficient of self-improvement $κ$ as the Lie derivative of the capability functional along this flow. The central contribution of this work is the derivation of the Variance Inequality, a spectral condition that is sufficient (under mild regularity) for the stability of self-improvement. We show that a sufficient condition for $κ> 0$ is that, up to curvature and step-size effects, the combined noise of generation and verification must be small enough. We then apply this formalism to unify the recent literature on Language Self-Play (LSP), Self-Correction, and Synthetic Data bootstrapping. We demonstrate that architectures such as STaR, SPIN, Reflexion, GANs and AlphaZero are specific topological realizations of the GVU operator that satisfy the Variance Inequality through filtration, adversarial discrimination, or grounding in formal systems.

• The paper shows that the GVU operator decomposes self-play into generator, verifier, and updater roles to drive sustained self-improvement.
• It establishes a Variance Inequality as the spectral condition for effective capability growth by balancing noise and bias in the self-improvement loop.
• The study details both diagonal and ensemble architectures, revealing that ensemble methods reduce verification noise and enhance performance.

Self-Improving AI Agents through Self-Play: A Unified Geometric and Spectral Framework

Overview

This work introduces a rigorous, moduli-theoretic formalism for agent self-improvement in AI, extending prior psychometric battery models to dynamical systems via the Generator-Verifier-Updater (GVU) operator. It addresses the critical, yet unresolved, issue of ignition: whether and how an agent can autonomously convert computational resources into sustainable capability gains. The paper synthesizes a vast swath of self-play, self-correction, and synthetic data literature under a geometric theory, anchored by a spectral Variance Inequality that precisely characterizes the regime where self-improvement is possible.

The GVU Operator as Unified Self-Improvement Mechanism

At the heart of the framework is the GVU operator, a canonical decomposition of the self-improvement loop into three roles: Generator (sampling behavior), Verifier (internal scoring), and Updater (first-order policy adjustment). The agent is modeled as a flow $\nu_r$ on a parameter manifold $\Theta$, with capability measured via a functional $F(\theta) = \Phi_{\mathcal{B} \circ \rho_{\mathcal{B}}(\theta)$ induced by an external psychometric battery. The Lie derivative of $F$ along the agent's trajectory, denoted by $\kappa$, operationalizes the notion of self-improvement.

The universality theorem shows that any rational, sample-driven self-update admits a score-based GVU representation in REINFORCE form. In detail, for a smooth statistical manifold endowed with Fisher information, any first-order update can be realized as the expected policy gradient over a scalar potential $V_\theta$. This places RL, supervised finetuning, adversarial training, and synthetic bootstrapping under a common algebraic structure.

Diagonal and Ensemble GVU: Noise and Bias Structure

The paper distinguishes between diagonal GVU (where the same model generates, verifies, and updates) and ensemble GVU (where these roles are separated among multiple agents). The diagonal regime is prone to self-confirmation and spectral entanglement—generator and verifier share noise statistics, leading to stagnation or collapse (the Hallucination Barrier). Ensemble GVU structures, such as councils of LLMs, empirically reduce verification noise via aggregation, improving signal-to-noise ratio and successively increasing the local self-improvement slope $\kappa$.

The Variance Inequality: Spectral Condition for Self-Improvement

A central mathematical contribution is the derivation of the Variance Inequality, a quantitative spectral condition linking expected capability gain to alignment ($\rho$), step size ($\eta$), curvature ($L$), and the joint SNRs of the generator and verifier. Explicitly, for expected improvement, it is required that

$$\rho \|g^*\|^2 > \frac{\eta L}{2} \left( \rho^2 \|g^*\|^2 + \sigma_{\mathcal{G}}^2 + \sigma_{\mathcal{V}}^2 \right)$$

where $\sigma_{\mathcal{G}}^2$ and $\sigma_{\mathcal{V}}^2$ are variances in generation and verification respectively. The sufficiency and necessity of a non-trivial verifier ($V_\theta$ not constant) is mathematically proven, and the Hallucination Barrier is identified: when verification noise matches generation noise, the loop cannot produce sustained self-improvement ($\kappa > 0$).

Geometric Interpretation and Moduli Topologies

The agent’s parameter space is treated as a statistical manifold with Fisher information geometry, while the moduli space stratifies batteries and tasks into fibers (Sociality, Planning, Embodiment, Alignment, Recursive, Synthetic, Critic-less). The paper interprets the GVU drift as a vector field, with geometric alignment measured via the Fisher angle between the update mean $v(\theta)$ and the true gradient $g^*$, and spectral stability dictated by variance and step size. This lens is applied to re-structure the landscape of self-improving architectures.

Topological Realizations of GVU in Literature

Multiple self-improvement architectures are subsumed under specific GVU topologies:

• AlphaZero (Sociality): Generator samples self-play games, verifier returns deterministic outcomes, updater performs supervised or RL policy adjustment; high-SNR verification ensures sustained $\kappa$.
• Adversarial Self-Play (SPIN/LSP, GANs): Verifier discriminators are easier to train than generators, resulting in improved verification SNR relative to generation.
• STaR, PRMs (Planning): Deterministic or dense process supervision ensures low verification variance, enabling robust reasoning bootstrapping.
• Self-Debugging Code Agents (Embodiment): Compilation and execution against unit tests provide oracle-like low-noise verification.
• Reflexion, Debate (Recursive): Model acts as its own critic. High risk of spectral failure unless ensemble or interface asymmetries are introduced.
• RLHF, Constitutional AI (Alignment): Parametric verifiers trained on human or AI-judged data; ensemble or group-based verification reduces variance and expands stable improvement regimes.
• Self-Instruct (Synthetic): Diagonal GVU regime, vulnerable to the Hallucination Barrier, unless external heuristics or strict filters are introduced.
• GRPO (Critic-less): Group-based normalization constrains update variance, analytically reducing verification noise, and improving stepsize stability.

Numerical scaling laws are shown for ensemble judges and group normalization, including $1/N$ reductions in verification noise and proportional growth of admissible stepsize, directly governed by the spectral theory.

Formal Definition and Dynamics of AI Slop

AI slop—ubiquitous low-quality outputs in LLMs—is rigorously defined as regions where the internal Verifier highly ranks outputs that score poorly on the external battery. The mass of such slop is mathematically quantified and shown to increase when the Variance Inequality fails, leading to collapse toward high-entropy, low-capability attractors.

Practical Implications and Operationalization

The theory prescribes explicit design levers:

• Strengthen verification (increase $\mathrm{SNR}(\mathcal{V})$) via ensembles, oracles, group normalization, or stricter prompts.
• Optimize step size and curvature regime, leveraging geometric insights for natural-gradient flows.

An operational protocol is proposed for empirically measuring self-improvement rate $\hat{\kappa}$ under fixed compute, explicitly separating static knowledge from autonomous improvement.

Agents aiming for AGI must demonstrate $\hat{\kappa} > 0$ uniformly across diverse moduli fibers, with universal, robust verification mechanisms.

Conclusion

This paper fuses self-play, adversarial, process, synthetic, and alignment-based improvement paradigms into a unified spectral-geometric theory, proving that autonomous capability growth is mathematically and operationally bounded by the spectral structure of the GVU operator. Successful self-improving designs systematically exploit topologies and interfaces where verification is spectrally favored, with empirical scaling laws quantifying their advantage. Realizing true AGI requires closing the universal verification gap across the moduli space, sustaining positive $\kappa$-flow under practical resource constraints.

Reference: "Self-Improving AI Agents through Self-Play" (2512.02731)