MetaGAI: A Large-Scale and High-Quality Benchmark for Generative AI Model and Data Card Generation

Abstract: The rapid proliferation of Generative AI necessitates rigorous documentation standards for transparency and governance. However, manual creation of Model and Data Cards is not scalable, while automated approaches lack large-scale, high-fidelity benchmarks for systematic evaluation. We introduce MetaGAI, a comprehensive benchmark comprising 2,541 verified document triplets constructed through semantic triangulation of academic papers, GitHub repositories, and Hugging Face artifacts. Unlike prior single-source datasets, MetaGAI employs a multi-agent framework with specialized Retriever, Generator, and Editor agents, validated through four-dimensional human-in-the-loop assessment, including human evaluation of editor-refined ground truth. We establish a robust evaluation protocol combining automated metrics with validated LLM-as-a-Judge frameworks. Extensive analysis reveals that sparse Mixture-of-Experts architectures achieve superior cost-quality efficiency, while a fundamental trade-off exists between faithfulness and completeness. MetaGAI provides a foundational testbed for benchmarking, training, and analyzing automated Model and Data Card generation methods at scale. Our data and code are available at: https://github.com/haoxuan-unt2024/MetaGAI-Benchmark.

• The paper introduces MetaGAI, a benchmark that automates model and data card generation via a multi-agent framework, ensuring high-fidelity documentation.
• It leverages retriever, generator, and chief editor agents to synthesize multi-source evidence, improving output quality by 15-20% over unrefined drafts.
• The study demonstrates MoE models' superior cost-quality trade-off while highlighting challenges like long-context reasoning and information sparsity.

MetaGAI: Establishing a Large-Scale Benchmark for Automated Model and Data Card Generation

Motivation and Benchmark Construction

The exponential growth of Generative AI (GenAI) models and datasets, alongside increased deployment in critical applications, has created urgent demands for scalable, high-fidelity documentation practices. Existing manual approaches to Model and Data Card curation suffer from insufficient scope, inconsistency, and incompleteness, especially outside industrial settings. Automated generation methods have emerged but have been obstructed by the lack of rigorous, large-scale benchmarks, limiting systematic progress. MetaGAI addresses this gap by constructing a benchmark of 2,541 document triplets where each instance groups an academic paper, a corresponding GitHub repository, and an associated Hugging Face artifact, verified through multi-step semantic triangulation to ensure referential integrity.

The construction pipeline leverages a multi-agent framework: Retriever Agents segment and align source documents with a detailed schema, Generator Agents synthesize diverse candidate drafts for each metadata field, and a Chief Editor Agent consolidates, validates, and refines these drafts against the entirety of context, integrating explicit human-in-the-loop validation at multiple stages.

Figure 1: An example of MetaGAI’s automated multi-source card generation using both scientific and artifact sources for maximal metadata fidelity.

Figure 2: Detailed depiction of the multi-agent, multi-source construction pipeline with four-tiered validation, including expert human adjudication.

Scope, Taxonomy, and Corpus Statistics

MetaGAI’s structured schema extends the Model Card and Data Card templates to capture GenAI-specific attributes—such as generative capabilities, safety, ethical dimensions, privacy, dataset lifecycle, and technical entity details. Card ground truth is synthesized by integrating signals from the arXiv paper, implementation from GitHub, and deployment/availability information from Hugging Face, mitigating the well-documented absence of implementation specifics in research manuscripts.

The corpus is constructed through canonical filtering and semantic verification, resulting in broad coverage across arXiv domains and temporal trends (2019–2025), and is distributed to maximize granularity and complementarity across sources:

Figure 3: Distribution of MetaGAI entries by year/domain (top) and relative wordcount density per source (bottom).

Multi-Agent Pipeline: Retrieval, Generation, and Synthesis

MetaGAI's benchmark generation process is formalized as mapping an unstructured scientific artifact $\mathcal{P}$ to a standardized, multi-field card $\mathcal{C}$, with the ground truth $\mathcal{C}_{GT}$ derived from triplet evidence $(\mathcal{P}, \mathcal{G}, \mathcal{H})$. Evaluation, however, is restricted to $\mathcal{P}$ input in order to reflect real-world constraints, presenting a deliberate asymmetry between reference and candidate conditions.

• Retriever Agent: Employs chunk-level, generative schema alignment, outperforming discriminative rerankers (P@1 = 0.635 for Qwen3-30B-A3B-Instruct) in mapping unstructured text to card fields.
• Generator Ensemble: Diverse architectures (OLMo-3-7B, Llama-3.1-8B, Qwen2.5-7B) generate field-wise drafts with accompanying provenance and quoted evidence, ensuring coverage of varied interpretations from the source.
• Chief Editor Agent: Performs cross-document consolidation and evidence-based merging, mitigating hallucinations and incomplete extractions by verifying content prominence and cross-field consistency.

Benchmark Validation: Human-in-the-Loop and Statistical Analysis

MetaGAI’s ground truth quality is established through exhaustive validation protocols, addressing critical reliability criteria:

• Retrieval validation confirms generative LLM superiority for schema alignment.
• Generator divergence analysis combines mean-variance metrics (BERTScore) to reveal that high-consensus fields reflect evidence sparsity, while information-rich fields benefit from architectural diversity among generators.

  Figure 4: Pairwise semantic similarity among generator outputs, showing variance as a function of field evidence density.

Human annotation and LLM-as-a-Judge scoring on a representative sample (N=100) establish a $15$–$20\%$ increase in card quality (Likert scores >4.0) for editor-refined outputs relative to unrefined generator drafts, demonstrating that the multi-agent design elevates the ground truth above baseline noise.

Empirical Assessment of Automated Generation

Evaluations are executed in a zero-shot, paper-only regime, mirroring real-world use cases. Models compared include dense transformer and Mixture-of-Experts (MoE) open-weight LLMs, as well as leading proprietary systems (GPT-5.x, Gemini-2.5).

Key empirical findings:

• MoE Superiority: Sparse MoE architectures, when scaled to 30B+ parameters (e.g., Qwen3-30B-A3B-Instruct), surpass dense models both in absolute quality (editor- and judge-rated >4.5) and in cost-efficiency.
• Cost-Quality Pareto Frontier: Open-weight MoE models dominate the cost/quality trade-off surface, achieving state-of-the-art performance at a small fraction of proprietary model inference cost.

Figure 5: Pareto-optimal cost-efficiency curve for card generation, highlighting the dominance of open-source MoE models over closed-source options.

• Metric Invalidity: Lexical similarity metrics (ROUGE-L, BERTScore) are found to be unreliable, inversely correlating with editordefined semantic quality. Traditional field completeness is largely orthogonal to judged qualitative utility.
• Faithfulness–Completeness Trade-off: Even top-performing models precisely reproduce supported fields but systematically omit up to 20% of fields due to information sparsity and long-context retrieval failures.

Analysis of Cognitive and Data Constraints

MetaGAI identifies systemic challenges faced by current LLMs when generating data-centric documentation:

• Universal Data Card Difficulty: Models uniformly underperform on Data Card fields, attributable to the high abstraction and information sparsity of real-world dataset documentation, in contrast to better-populated Model Card schemas.

Figure 6: Fine-grained performance across card fields for all model baselines, demonstrating strong degradation in sparsely-specified, abstract fields.

• Information Sparsity as Primary Bottleneck: Regression analysis ($R^2=0.31,\ p=0.011$) confirms that card field coverage is driven by the density of input evidence rather than by retrieval failure or context truncation.

  Figure 7: Linear relationship between ground truth information density and generation success, underscoring the challenge of abstract metadata synthesis.

• Lexical Divergence: Statistical comparison (Log-Odds Ratio) of generated outputs vs. ground truth reveals that human-edited cards uniquely capture high-value technical entities, while baseline generations are biased towards narrative language and structure hallucinations.

Figure 8: Systematic divergence in lexical content between autogenerated outputs and edited ground truth, emphasizing the value of explicit synthesis and entity verification.

Implications and Future Research Directions

Practically, MetaGAI enables the systematic evaluation and fine-tuning of LLMs for trustable, high-fidelity metadata synthesis across thousands of GenAI artifacts. MoE scaling emerges as the most cost-effective, high-quality method for structured generation at scale. Theoretically, MetaGAI highlights fundamental limitations in long-context reasoning and the intractability of document field completion under information-sparse conditions, motivating research on:

• Advanced long-context retrieval and reasoning.
• Techniques for synthetic information augmentation in under-documented domains.
• Integration of multimodal and graph-based evidence chains to bridge the abstraction gap in documentation.
• Robust validation and human-in-the-loop oversight frameworks to ensure safety in automated metadata generation.

Conclusion

MetaGAI represents a significant advance in the infrastructure underpinning automated GenAI documentation: it delivers scale, schema richness, multi-source referential integrity, and trusted ground truth via expert-oriented construction and validation. Benchmarking with MetaGAI provides critical guidance for both research and deployment in responsible AI, exposing key limitations of current generation paradigms and sharpening the focus for future methodological advances (2604.23539).