The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence

Abstract: The metanym game is a competitive word game for LLMs that measures structural intelligence against established cognitive-science constructs. No content is given in advance; the contestants create all of it -- a new kind of analogy test, analogical production falsifiable sentence by sentence, with no fixed test set to leak into training (contamination-resistant by construction). In the council-of-peers benchmark, the contestants also rate each other's creations. We introduce the first spectral solution, to our knowledge, to the wicked problem of benchmarking LLMs' factual accuracy without golden keys or oracle models: one singular value decomposition of the evaluators' ratings matrix yields their competence as both generators and judges of true statements at once. Competence on the subjective criteria comes from each judge's rating consistency as the yardstick shifts. The factual rating correlates with GPQA Diamond at Pearson r = 0.92. Scored separately, making and judging dissociate -- judging is the scarcer skill: the strongest generators are middling judges, the sharpest judge a mid-pack generator. To scale, the strongest players form a council that does the official benchmarking; its seats are contestable -- a stronger model earns one on the benchmark's own rating. The benchmark is entirely self-contained and self-consistent, a stable gauge over time.

• The paper introduces a novel protocol where LLMs both generate and evaluate analogical outputs, shifting from fixed test sets to dynamic, bias-resistant benchmarking.
• It employs spectral (SVD-based) aggregation to quantify factual accuracy and structural diversity, ensuring robust self-administered evaluation.
• Empirical results reveal a strong dissociation between generation and judgment skills among models, substantiating the benchmark’s validity and reliability.

The Metanym Game: A Self-Consistent LLM Peer Benchmark for Structural Intelligence

Overview and Motivation

The metanym game introduces a fundamentally new paradigm for benchmarking the structural intelligence of LLMs, departing from conventional approaches that rely on fixed test sets, gold labels, or human raters. Instead, this protocol tasks models with collaboratively both generating and evaluating each other’s analogy-rich outputs, operationalizing abstraction and analogy production as a falsifiable, contamination-resistant task. The core objective is to assess LLMs’ ability to instantiate and recognize deep, context-spanning structural isomorphisms without any dependency on external data or authorities.

The Metanym Game Protocol

Task Design

The metanym game employs a production-based test in which each participant LLM constructs "archetypal contexts"—abstract templates encoding covariant relationships among entities independent of surface domain. These templates contain slots for “metanyms”: domain-specific instantiations that, when swapped, transpose the template from one field to another while preserving structural relations. Success criteria require that each sentence in the substituted output must be verifiably true in the target domain, enabling granular falsification and breaking away from holistic or holistic analogy tests.

Both the construction of these templates and their cross-domain instantiations are fully unsupervised; no reference set or external solution key exists. Each participating model additionally evaluates all others’ outputs along six axes: factual accuracy (per parallel context), beauty, intelligence, instantiation distinctness, impressive length, and structural diversity.

Contamination Robustness

Unlike traditional benchmarks, the metanym game is natively resistant to test data contamination. All content—templates, instantiations, and the specific analogical mappings—are produced dynamically on each run, eliminating “test set leakage” as a confound. This feature is critical for accurately measuring progress in large-scale, continually-updated LLMs where static benchmarks quickly saturate or become obsolete.

Self-Consistent and Self-Contained Evaluation

A key innovation is the use of models as both generators and scorers, introducing a spectral (SVD-based) approach for the factual rating aggregation. Agreement among evaluators is analyzed via singular value decomposition of the ratings matrix—competence is inferred by alignment with the dominant axis of (leniency-removed) consensus, rather than any preordained gold label. For subjective criteria, anchor-shift consistency (Pearson correlation of per-axis scores under calibration-point shifts) quantifies an evaluator’s internal criterion reliability, obviating the necessity for human intervention or anchoring.

Council membership (i.e., official evaluators) is determined via dual competence gates: factual error detection above the “inert band” and robust cross-anchor reliability for the non-factual axes. Only models meeting both criteria are admitted as trusted judges; others may participate as generators only.

Empirical Results and Discriminative Power

Council Selection and Tiers

Twelve leading LLMs (Anthropic Claude Opus 4.x, Claude Sonnet 4, Google Gemini 3.1 Pro, Gemini 2.5 Flash, and OpenAI GPT-4.1/-4o variants) were evaluated in a blinded, cross-comparative protocol. The SVD-based competency estimator cleanly separated the field into three capability tiers (Anthropic Claude 4.x top, Gemini/GPT-4.1-mid, OpenAI 4o/4.1 nano at floor) with bootstrapped CIs. Five models (three Claude Opus and two Gemini) met both factual and criterion-reliability requirements and formed the initial council, with others excluded for either factual inertness or consistency instability.

Generation vs. Judging Dissociation

Crucially, and in contradiction to prevailing assumptions in prior unsupervised peer-evaluation protocols [Ning et al., (Ning et al., 2024); Zhang et al., (Zhang et al., 19 Mar 2025)], the results show a strong dissociation between generation and judgment skills:

• The best factual judges are not necessarily strong generators. Gemini 3.1 Pro was the top factual judge but a mid-pack generator; Sonnet-4 generated near the top but scored poorly as a judge.
• Aggregate factual scoring (mean of generation and evaluation factual competencies) correlated strongly with independent external benchmarks (GPQA Diamond; Pearson $r = 0.92$ across twelve models), providing robust evidence that the key-free spectral estimator tracks real capability.

Reliability of Self-Administered Judgment

Analysis of single-vendor and multi-vendor subpanels demonstrated that council-derived factual ratings are stable and not merely artifacts of vendor-specific self-preference. Removal of any vendor’s models as judges negligibly perturbed rankings (minimum Spearman $\geq 0.96$ vs. full panel), and the dominant agreement axis persisted, supporting the hypothesis that agreement is driven by competence, not brand-aligned bias.

Numerical Results Alignment

Figure 1: Combined factual rating $\tfrac{1}{2}(E^F + G^F)$ (key-free, anchored 1--10) vs. self-administered GPQA Diamond accuracy, yielding Pearson $r=0.92$.

The top-tier models, as determined by the metanym benchmark, correspond almost exactly with those scoring highest on GPQA Diamond. This convergence strongly substantiates the validity of the metanym approach as a measure of structural and factual intelligence.

Theoretical and Methodological Innovations

Production-Task–Driven Analogy Testing

Unlike recognition-based tasks used in BIG-Bench or ARC [Chollet, (Chollet, 2019)], the metanym game shifts the benchmark to an open-ended production task. This aligns the benchmark’s demands with cognitive science theory regarding abstraction, analogy, and higher-order relational reasoning (Gentner 1983, Penn et al. 2008), operationalizing productional, falsifiable analogy mapping as a multidomain, sentence-level test.

Spectral Estimation Without External Keys

The use of SVD/spectral aggregation for inferring both generator and judge competence from a single ratings matrix is novel in the LLM evaluation literature. Previous approaches [PiCO, UPME] rely on binarized judgments or optimization-based consistency, but not on a graded, theoretically-grounded consensus extraction. This approach allows simultaneous, contamination-resistant measurement of generative and evaluative strength, a property not available in previous unsupervised peer-evaluation schemes.

Anchor Consistency and Calibration

For subjective axes, the consistency of evaluations is validated by anchor sweeping, embedding judge reliability directly into the protocol. This corrects for the pitfall where agreement becomes indistinguishable from conformity in aesthetic or preference-based tasks, ensuring that council membership requires not only consensus-seeking but also internal standard stability.

Implications and Future Directions

Sustainability and Contestability

Because the protocol is self-contained, self-consistent, and reproducible (T=0, no reasoning, tools off), the benchmark can scale by simple addition: any new model can be submitted, scored by the council against the fixed anchor. The contestable “chair” mechanism allows new models to join or supplant council seats by both generation and evaluation competence, preserving continuous competition and up-to-date relevance as LLM capabilities evolve.

Toward Structural Intelligence Metrics

The metanym game isolates and quantifies abstraction and analogy—the elements argued to be central to higher-order intelligence—free from test leakage and external label requirements. This allows for benchmarking of cognitive dimensions foundational to AGI, including not only fluid and crystallized intelligence but also convergent and divergent reasoning, with direct ties to mechanistic and representational analyses.

Prospects for Mechanistic Insights

An open-weight panel would allow control for parametric and provider-based confounds. Extending the protocol to probe whether archetypal contexts occupy low-dimensional subspaces of model hidden states could bridge benchmark performance and mechanistic interpretability, potentially providing an objective grounding for subjective criteria.

Conclusion

The metanym game establishes the first self-sustaining, self-consistent, contamination-resistant, production-task-centric benchmark of structural intelligence in LLMs. Its spectral aggregation methodology convincingly tracks both generation and judgment competence, revealing a heretofore unquantified divergence between making and evaluating cross-domain analogies. Ongoing developments may enable the benchmark to become fully self-sustaining—capable of external ground validation, iterative panel improvement, and eventual integration with mechanistic studies of representation and abstraction in deep models. This framework sets a precedent for next-generation AI evaluation standards centered on abstraction, analogy, and the dynamical interplay between generation and judgment.