The 2025 Foundation Model Transparency Index
Abstract: Foundation model developers are among the world's most important companies. As these companies become increasingly consequential, how do their transparency practices evolve? The 2025 Foundation Model Transparency Index is the third edition of an annual effort to characterize and quantify the transparency of foundation model developers. The 2025 FMTI introduces new indicators related to data acquisition, usage data, and monitoring and evaluates companies like Alibaba, DeepSeek, and xAI for the first time. The 2024 FMTI reported that transparency was improving, but the 2025 FMTI finds this progress has deteriorated: the average score out of 100 fell from 58 in 2024 to 40 in 2025. Companies are most opaque about their training data and training compute as well as the post-deployment usage and impact of their flagship models. In spite of this general trend, IBM stands out as a positive outlier, scoring 95, in contrast to the lowest scorers, xAI and Midjourney, at just 14. The five members of the Frontier Model Forum we score end up in the middle of the Index: we posit that these companies avoid reputational harms from low scores but lack incentives to be transparency leaders. As policymakers around the world increasingly mandate certain types of transparency, this work reveals the current state of transparency for foundation model developers, how it may change given newly enacted policy, and where more aggressive policy interventions are necessary to address critical information deficits.
Summary
- The paper introduces a comprehensive benchmark of 100 indicators across upstream, model, and downstream domains to assess FM transparency worldwide.
- It reveals a significant decline in transparency—from an average score of 58 in 2024 to 40 in 2025—highlighting marked opacity and regional disparities.
- Methodological advancements include granular subdomain analysis, hybrid human-LLM evaluations, and the first cross-regional comparative study.
The 2025 Foundation Model Transparency Index: Methodology, Findings, and Implications
Overview and Motivation
The 2025 Foundation Model Transparency Index (FMTI) (2512.10169) introduces the third comprehensive benchmark for quantifying and analyzing the transparency practices of major foundation model (FM) developers globally. As foundation models become central to economic infrastructure and societal functions, opacity in their development and deployment processes presents significant challenges for governance, accountability, and risk mitigation. This paper systematically measures FM developer transparency across upstream (data, compute), model (architecture, evaluation, risk), and downstream (deployment, societal impact, governance) aspects, updating and strengthening methodologies to align with current technical and regulatory trends.
Methodological Expansion and Novelty
The 2025 FMTI incorporates substantial methodological advances:
- Expanded Indicator Set: The index now comprises 100 indicators, with 55 new indicators focusing on data acquisition, downstream usage, and post-deployment monitoring, reflecting evolving risks and regulatory attention.
- Granular Subdomain Coverage: Indicators are partitioned into 18 subdomains along the foundation model supply chain, rigorously addressing organizational resources, model properties, and downstream impacts.
- Enhanced Scoring Procedures: Evaluation involves both company-prepared and FMTI-prepared transparency reports, with the latter utilizing hybrid human-expert and LLM-based retrieval when direct company cooperation is absent.
- Global Scope: The 2025 assessment extends to major Chinese and European developers, providing the first cross-regional comparative transparency analysis.
Figure 1: The 18 subdomains in the FMTI 2025 indicators parallel the foundation model supply chain, bridging resources, model specifics, and societal impact.
Figure 2: The 100 indicators for the 2025 FMTI, organized into upstream, model, and downstream domains.
Figure 3: Substantial indicator set revision in 2025—55 new indicators, 9 retained, 36 heavily modified, split, or merged relative to 2024.
Key Findings: Declining Transparency and Structural Patterns
Aggregate Trends
Transparency Regression: Contrary to the positive trajectory noted in 2024, overall transparency dropped sharply; the mean score decreased from 58 (2024) to 40 out of 100 (2025), evidencing marked organizational retrenchment. The entire distribution of developers shifted toward increased opacity, particularly among US-based and frontier companies.
Figure 4: Domain-level transparency scores per company in 2025. Notably, IBM stands as a high-scoring outlier.
IBM as an Exception: IBM achieved an unprecedented 95/100, including six indicators uniquely disclosed among all developers. By contrast, xAI and Midjourney reached an index nadir at 14/100.
Transparency Clusters: Developers stratify into three clusters—top scorers (IBM, Writer, AI21 Labs; avg=78), intermediate performers (e.g., Anthropic, Google, OpenAI; avg=36), and low scorers (Mistral, xAI, Midjourney; avg=15).
Domain-Specific Insights
Upstream (Data, Compute)
- Extreme opacity persists in disclosing training data sources, data properties, and compute expenditures. Most companies obfuscate or omit core details regarding dataset composition, compute hardware, energy, and financial outlay, citing competitive and legal constraints.
- Only the compute provider is widely disclosed (9/13), with IBM and Writer being the sole entities transparently reporting complete compute and environmental impact metrics.
Model (Capabilities, Risks, Release)
- Companies are comparatively more forthcoming regarding model capabilities and architectures, yet such disclosures often lack methodological transparency and reproducibility.
- Risks are less thoroughly documented than capabilities, and only IBM’s risk evaluation achieves external reproducibility standards. No developer provides detailed train–test overlap disclosure.
- Release and versioning protocol transparency is high (average 69%), but variant across regions and business models.
Downstream (Usage, Impact, Policy, Accountability)
- Acceptable use policies and model behavior guidelines demonstrate broad availability; downstream impact, usage data, and feedback/incident reporting mechanisms remain highly opaque.
- Accountability mechanisms, including whistleblower protections and government reporting, are documented by a minority.
Figure 5: Transparency scores by major subdomains, evidencing heterogeneous priorities and areas of systemic opacity.
Stratified and Longitudinal Analyses
Openness and Business Model
Figure 6: Open-weight model developers have higher average transparency, but several influential open-weight developers exhibit significant opacity.
- Open-Weight Versus Closed-Weight: Openness correlates with, but does not guarantee, transparency. While IBM and AI21 Labs (open-weight) display robust transparency, high-impact open-weight developers like Meta, Alibaba, and DeepSeek have subpar transparency profiles.
Enterprise Versus Consumer-Oriented
Figure 7: Business-to-business (B2B) developers are consistently more transparent than consumer or hybrid entities, particularly for upstream disclosures.
- Enterprise-Focused Developers: B2B orientations (IBM, Writer, AI21 Labs) strongly associate with superior transparency, both at aggregate and upstream levels, compared to B2C or hybrid models.
Regional and Industry Groupings
Figure 8: US-headquartered developers slightly outperform non-US peers, but include both the highest and lowest index scores.
Figure 9: Frontier Model Forum (FMF) members occupy middling positions, suggesting peer-group incentive alignment yields convergence but not leadership in transparency.
Figure 10: EU AI Act Code of Practice signatories marginally outperform non-signatories, driven by new regulatory expectations.
Method of Reporting
Figure 11: Companies submitting their own transparency reports score nearly double those scored solely on external information.
Cross-Year Regression
Figure 12: Cross-year domain scores; regression in transparency from 2024 to 2025 is driven predominantly by upstream and downstream domains.
Figure 13: Longitudinal transparency trends from 2023 to 2025, with major volatility by company.
- Transparency progress observed 2023→2024 is reversed in 2025, with multiple instances of outright regression (e.g., Meta and OpenAI dropping from first and second in 2023 cohort to last and second-to-last, respectively, in 2025).
- Only AI21 Labs, Amazon, and Writer demonstrate sustained or improved transparency profiles over three years.
Strong Numerical Outcomes, Contradictory Trends, and Policy Implications
- Average index score: 40/100 (2025), down from 58/100 (2024)
- IBM: 95/100—highest historically recorded
- xAI, Midjourney: 14/100—second-lowest all-time
- 55/100 indicators completely new in 2025
- Open-weight developers: bifurcated—some among highest, others far below average
- B2B model developers: substantially higher transparency than any other group
Key contradictory outcome: Despite policy and public pressure, leading companies have systematically reduced transparency, often removing previously available details from public artifacts (e.g., Model Cards, System Cards, privacy documentation).
Theoretical and Practical Implications
Theoretical Impact: The FMTI's elaboration of indicators establishes a taxonomy for transparency that may directly inform regulatory schema and industry standards. The fine-grained mapping of transparency “deficits” produces an empirical basis for reasoning about where market forces alone are insufficient to drive disclosure, motivating work on measuring socio-technical externalities and aligning incentives.
Policy and Industry Implications: As major governance frameworks (e.g., EU AI Act, US state regulations) mature, FMTI evidences the difficulty in achieving substantive transparency gains through voluntary or reputationally-motivated action. The clear link between direct engagement (company-prepared reports) and high FMTI scores underscores the limited effectiveness of external investigation alone. Furthermore, the regulatory-mandated requirement to disclose training data summaries in the EU is poised to create measurable improvements, with future editions of the FMTI serving as a critical longitudinal monitor.
Prospects for AI Development: Extensive opacity regarding data provenance, compute resources, and deployment routes not only hinders reproducibility and external auditability but undermines collective progress on robust model evaluation, societal risk assessment, and the advancement of best practices for safety/impact monitoring.
Conclusion
The 2025 Foundation Model Transparency Index (2512.10169) rigorously demonstrates eroding transparency among leading foundation model developers despite increased societal, academic, and regulatory focus. The findings foreground the structural roots of transparency deficits, convergent behaviors among peer groups, and the sharp contrast between organization types. The Index's sustained measurement framework not only advances the methodological frontier for transparency evaluation but provides a crucial instrument for regulatory authorities, third-party auditors, and researchers navigating the complex socio-technical landscapes of foundation models. Future work must attend to changing regulatory environments, further automate the evaluation pipeline, and develop interventions targeting the root causes of persistent opacity.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper is about a “report card” for big AI companies. It measures how open and honest they are about how they build and run their most important AI systems, called foundation models (think of chatbots like ChatGPT or image tools like Midjourney). The report card is called the Foundation Model Transparency Index (FMTI), and 2025 is its third year.
The main message: transparency got worse in 2025. On average, companies scored 40 out of 100 (down from 58 in 2024). IBM did best with 95, while xAI and Midjourney were lowest at 14. The areas companies are most secretive about are the data they use to train their models, how much computer power they use, and what happens after the model is released (how it’s used and its impact).
Objectives and Research Questions
The paper asks simple but important questions:
- How open are AI companies about what goes into building their models, what the models can do, and what happens after they’re released?
- Which parts of the AI “pipeline” are most hidden: the inputs (data and computing), the model itself, or the outcomes (usage and impact)?
- Do certain kinds of companies (like those focused on business customers, or those that share model weights) act more transparently?
- How has transparency changed since 2023 and 2024?
- What should policymakers and the public do if key information is missing?
Methods and Approach
Think of the FMTI as a big checklist with 100 questions (called “indicators”) that cover three stages of the AI pipeline:
- Upstream: what happens before a model is built (data sources, how data is cleaned, computer hardware, energy use, and cost).
- Model: what the model is and how it’s released (architecture, access, capabilities, risks, safety fixes, release process).
- Downstream: what happens after release (who uses it, where it’s used, how risks are monitored, policies, accountability).
Here’s how the team ran the Index:
- They designed updated indicators for 2025 to better fit today’s AI world. For example, they added:
- Data acquisition methods (including top-5 sources for public, licensed, user, and synthetic data).
- Post-deployment monitoring (like bug bounties and incident reporting).
- Accountability (like oversight and whistleblower protections).
- Agent protocols and model theft prevention.
- They chose major AI companies to evaluate (including new ones like Alibaba, DeepSeek, and xAI).
- They gathered information from public sources and asked companies to submit transparency reports. Only 7 companies sent reports (down from 14 in 2024), so the team manually collected info for several others.
- They scored each company using the 100 indicators.
- They talked with companies to clarify details and encourage better disclosure.
The team also tried using AI agents to search for information. These agents helped find useful pieces, but the researchers still had to check everything by hand.
Key change in 2025: the bar was raised. For example, to say an evaluation is “reproducible,” companies now need to share code and prompts so outsiders can repeat the test. And rather than vague claims like “our model can write text,” companies are asked to list the specific abilities they optimized during training.
Main Findings and Why They Matter
- Overall transparency dropped: average score fell from 58 (2024) to 40 (2025), roughly back to 2023 levels (37).
- Biggest blind spots:
- Training data: where it came from, who was paid, and whether licensed content was used.
- Training compute: what hardware was used, how much energy was spent, and environmental impact.
- Post-release usage: how many users, who they are (consumers, businesses, governments), and what the model’s real-world effects are.
- Evaluation details: companies often share results but not enough to reproduce them; most don’t report whether training data overlaps with test data (like studying with the same questions as the exam).
- Standout companies and groups:
- IBM scored highest at 95 and disclosed several items no one else did.
- Top group: IBM, Writer, AI21 Labs (average 78).
- Middle group: Anthropic, Google, Amazon, OpenAI, DeepSeek, Meta, Alibaba (average 36).
- Bottom group: Mistral, Midjourney, xAI (average 15).
- Open-weight model developers tend to be more transparent, but it’s mixed: IBM and AI21 were strong, while DeepSeek, Meta, and Alibaba were relatively opaque.
- Enterprise-focused (business) companies (IBM, AI21 Labs, Writer, Amazon) are more transparent than consumer-focused ones.
- Companies that wrote their own transparency reports and engaged with the FMTI scored higher.
- Signatories to the EU AI Act’s Code of Practice scored slightly above non-signatories.
- Members of the Frontier Model Forum (Amazon, Anthropic, Google, Meta, OpenAI) landed in the middle—likely trying to avoid very low scores without going the extra mile.
- Year-over-year changes:
- Many companies’ scores fell (for example, Meta dropped by about half; Mistral by more than two-thirds; OpenAI also decreased).
- AI21 Labs improved a lot since 2023 (from 25 to 66 in 2025).
These findings matter because transparency builds trust, helps prevent harm, and allows researchers, journalists, and policymakers to understand and improve AI’s impact on society.
Implications and Potential Impact
- Current market incentives are not enough to make companies fully open about the most important parts of AI (like training data and compute). Without stronger rules, these gaps will likely remain.
- Policymakers can use the indicators as a template for disclosure requirements, especially on:
- Data sources and licensing
- Energy and environmental impact
- Reproducible evaluations and train-test overlap
- Post-deployment monitoring and incident reporting
- Accountability and oversight
- The Index helps:
- Companies see where they can improve and how they compare to peers.
- Customers and investors choose more responsible partners.
- Journalists and researchers focus on the biggest information gaps.
- Long-term, the FMTI encourages healthier AI ecosystems by making it normal—and expected—for companies to be clear about how their models are built and used.
In short: the 2025 FMTI shows we need stronger transparency standards. Some companies are leading the way, but overall openness fell this year. Clear rules, better reporting, and reproducible methods can help AI grow in a safer, fairer, and more trustworthy direction.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a consolidated list of concrete gaps and unresolved questions the paper leaves open, framed to guide actionable future research and methodological improvements:
- Longitudinal comparability: Major indicator redesign in 2025 undermines score continuity with 2023–2024; no bridging study, back-scoring, or calibration to ensure trend validity across editions.
- Indicator weighting and aggregation: Rationale for equal weighting is not justified; no robustness analysis of rankings under alternative weight schemes reflecting stakeholder priorities.
- Scoring reliability and transparency: Inter-rater reliability, adjudication protocols, codebook granularity, and evidence citation practices are not reported; no confidence intervals or uncertainty estimates on scores.
- Nonresponse and participation bias: Only 7 of 23 companies submitted reports; others were scored via manual discovery, potentially penalizing non-participants and rewarding participants; no correction or sensitivity analysis for nonresponse bias.
- Cross-jurisdiction and language bias: First inclusion of Chinese firms raises translation and discovery challenges; methods for multilingual evidence collection and verification are not specified.
- Verification and auditability of disclosures: The Index accepts self-reports without a standardized audit process; no mechanism for third-party verification, spot checks, cryptographic proofs, or audit trails to detect misreporting.
- Awarding points for acknowledged information deficits: Granting credit when companies disclose they lack data due to third-party constraints may create perverse incentives; criteria for minimal acceptable disclosure (e.g., naming counterparties, contract clauses) need formalization and safeguards.
- “Top-5 data sources” criterion: No standardized, reproducible method to rank sources (by tokens, documents, bytes, or contribution to loss); absence of audit-friendly, privacy-preserving provenance reporting standards.
- Train–test overlap: No operational protocol to quantify overlap without releasing datasets; lack of standardized hashing, sampling audits, or overlap metrics that balance IP/privacy with transparency.
- Compute, energy, and emissions reporting: System boundaries (training vs development, location-based vs market-based emissions, PUE/WUE inclusion) and verification with cloud providers are not standardized; no guidance on reconciling provider-level and developer-level disclosures.
- Usage and impact metrics: Definitions (active users, requests, sessions), denominators, and cross-channel comparability are unspecified; no privacy-preserving analytics framework to enable standardized, auditable reporting.
- Downstream monitoring validity: The predictive validity of monitoring indicators is untested (e.g., whether higher “monitoring” scores correlate with fewer or better-managed incidents); no incident dataset or evaluation benchmark is used.
- External validity of FMTI: No evidence links higher transparency scores to better societal outcomes (fewer harms, improved governance, increased competition) or to reduced regulatory or safety incidents.
- Potential gaming of the Index: Requiring company-prepared transparency reports can invite tailored disclosures; no post-publication audit sampling, falsification checks, or penalties for inaccuracies.
- AI agent–assisted evidence gathering: The paper notes agent use but lacks a systematic evaluation (recall/precision, time savings, error analysis) or a reproducible pipeline for others to adopt.
- Scope across modalities: Indicator applicability to non-text modalities (vision, speech, multimodal, generative media) and agent systems is not clarified; modality-specific transparency needs remain under-specified.
- Flagship model selection: Criteria for choosing a single “flagship model” per company are unclear; no method to aggregate across multiple flagship models or weight company-level scores by usage or market relevance.
- Company vs model granularity: Some indicators are organization-level, others model-level; how mixed granularity is reconciled into a single score is not formalized.
- Policy impact analysis: Claims about the effects of the EU AI Act Code of Practice and other policies are descriptive; no causal design (e.g., difference-in-differences) tests whether regulation drives observed transparency differences.
- Open vs closed developer differences: Observed patterns are not disentangled from confounders (business model, client mix, jurisdiction, size); no multivariate analysis to identify drivers of transparency.
- Organizational governance indicators: Asking for organization charts and oversight mechanisms lacks quality/effectiveness measures (e.g., independence, escalation authority, resourcing) and verification procedures.
- Reproducibility standards: Requiring open code/prompts for evaluation reproducibility excludes feasible alternatives for closed-source settings (e.g., secure evaluation rooms, attested runs, reproducible artifacts without code).
- Handling partial compliance: No formal scoring for partial disclosures or uncertainty; lack of graded scoring or probabilistic assessments to reflect incomplete evidence.
- Environmental impacts beyond energy/carbon: Water use, e-waste, land footprint, and grid mix dynamics are not included; standardized measurement and verification protocols are needed.
- Financial transparency: “Cost of training” lacks standardized definitions (capex vs opex, amortization, data/licensing vs compute); no verification against financial statements or third-party records.
- Government use and commitments: Taxonomy and verification of government commitments and deployments are unspecified; no data source strategy for sensitive or non-public contracts.
- Acceptable Use Policy (AUP) localization: Methods to collect and verify geographic policy variations and enforcement differences are not described; comparability across locales is unclear.
- Independence of third-party evaluators: Criteria to define “independent” vs vendor-contracted evaluators (funding, conflicts of interest, prepublication rights) are not specified.
- Train-time vs development-time compute: Indicators distinguish these concepts but lack standardized allocation rules for shared compute, multi-tenant infrastructure, and mixed workloads.
- Data usage from users: Standards for notice/consent, opt-out rates, and quantifying the contribution of user data to model improvements are not defined; privacy-preserving reporting methods are needed.
- Participation decline from 2024 to 2025: Causes of reduced company participation are not analyzed; strategies to sustain engagement or mitigate selection effects are not proposed.
- Public artifact availability: Figures state “full specification … can be found at .”, but links/artifacts are absent in the text; open repositories (with DOIs), full indicator definitions, raw evidence, and scoring rationales are needed for replication.
- Jurisdictional comparability: How differing legal constraints (e.g., trade secrets, national security) are normalized across jurisdictions to ensure fair scoring is not described.
- Thresholds for “sufficient” disclosure: Several indicators reference sufficiency without explicit, testable thresholds; standardized, measurable criteria would reduce subjectivity.
- Incident reporting taxonomy: Post-deployment incident reporting lacks a standardized taxonomy and minimum fields (severity, root cause, remediation timeline) for comparability.
- Agent-protocol support indicator: No canonical taxonomy or compliance testing methodology for agent protocols; interoperability benchmarks and verification are needed.
- Small vs large developers: The Index does not adjust expectations for SMEs’ resource constraints; fairness and proportionality of requirements is unaddressed.
- Effects of transparency on competition: The hypothesis that transparency promotes competition is asserted but not empirically tested (e.g., entry rates, vendor switching, procurement outcomes).
- Security vs transparency trade-offs: No framework or empirical evidence is provided to balance disclosure benefits with security/commercial risks; need structured guidelines for safe disclosure.
- Train–test overlap vs IP constraints: Practical protocols to report overlap without releasing corpora (e.g., salted hashing, TDM exceptions) are unaddressed.
- Consistency across distribution channels: Reporting standards for usage, AUP enforcement, and model behavior across APIs, SDKs, hosted UIs, and on-prem deployments are unspecified.
Practical Applications
Practical Applications of the 2025 Foundation Model Transparency Index
Below are actionable applications grounded in the paper’s indicators, methods, and findings. Each item names likely sectors, the tools/products/workflows that could emerge, and key assumptions or dependencies that affect feasibility.
Immediate Applications
- Vendor due diligence and procurement scoring for AI systems
- Sectors: enterprise software, finance, government procurement, healthcare
- Tools/workflows: FMTI-aligned RFP templates; procurement checklists keyed to indicators (e.g., top-5 data sources, reproducible evaluation code, AUP enforcement metrics); contract clauses requiring indicator disclosures
- Assumptions/dependencies: Companies’ willingness and legal ability to disclose; buyers’ capacity to audit; alignment with regional regulations (e.g., GDPR, EU AI Act)
- Transparency report generation and disclosure pipelines inside AI companies
- Sectors: software, platforms, cloud providers
- Tools/workflows: internal “transparency report generator” templates mapped to FMTI 2025 subdomains; automated data provenance logging; reproducible eval release kits (code + prompts + configs); model release changelog/versioning systems
- Assumptions/dependencies: Access to internal artifacts; security/IP constraints; coordination across legal, policy, engineering
- Post-deployment monitoring and incident response programs
- Sectors: cybersecurity, healthcare, finance, public sector
- Tools/workflows: AI bug bounty programs; responsible disclosure and safe harbor policies; incident reporting portals; misuse dashboards; coordination channels with government CERTs
- Assumptions/dependencies: Legal safe harbor maturity; third-party engagement; standardized incident taxonomies
- Acceptable Use Policy (AUP) enforcement reporting modeled on social platforms
- Sectors: consumer apps, developer platforms, enterprise SaaS
- Tools/workflows: quarterly AUP enforcement transparency reports (violations, actions, appeal rates) across distribution channels (API, hosted UI, on-prem); geography-specific policy variations documented
- Assumptions/dependencies: Reliable enforcement telemetry; privacy-compliant reporting; scalable moderation workflows
- Usage data and impact disclosure dashboards for customers and regulators
- Sectors: enterprise AI, public agencies, education
- Tools/workflows: anonymized usage categorization dashboards; retention schedules; geographic usage statistics; breakdown of dependent products/services and user types (consumer/enterprise/government)
- Assumptions/dependencies: Aggregation and anonymization pipelines; compliance with privacy law; clear categorization schemas
- Cloud provider reporting APIs for compute, energy, and environmental impacts
- Sectors: cloud/infra, energy, sustainability reporting
- Tools/workflows: per-tenant metering APIs that expose FLOPs/epochs, energy (kWh), carbon (tCO2e), and water; exportable reports for customers to fulfill FMTI indicators
- Assumptions/dependencies: Accurate metering in cloud stacks; standardized accounting methods; contractual permissions to re-share data
- Reproducible benchmarking kits and train–test overlap checks
- Sectors: ML research, model evaluation, education
- Tools/workflows: open-source evaluation harnesses (code, prompts, seeds); documentation patterns for “minor” implementation details; train–test overlap heuristics and hashing-based checks for common corpora
- Assumptions/dependencies: Access to or summaries of training data properties; community maintenance of tools; willingness to publish prompts/code
- Agent ecosystem interoperability disclosures
- Sectors: software agents, dev platforms, robotics
- Tools/workflows: publish supported agent protocols (e.g., function/Tool APIs) and compatibility matrices; SDK stubs; conformance tests
- Assumptions/dependencies: Stable protocol definitions; appetite to interoperate vs. lock-in
- Model behavior policy (MBP) transparency
- Sectors: consumer AI, education, safety
- Tools/workflows: publish restricted behaviors, response characteristics, default system prompts, and policies on intermediate tokens (e.g., chain-of-thought); side-by-side examples in docs
- Assumptions/dependencies: Balancing safety with disclosure (e.g., jailbreak risk); legal review; consistent MBP enforcement
- Organizational accountability mechanisms
- Sectors: all AI-developing organizations
- Tools/workflows: pre-deployment oversight committees; whistleblower policies; public commitments to government codes of practice; named governance owners in org charts
- Assumptions/dependencies: Executive buy-in; alignment with regulatory commitments; protection from retaliation
- Semi-automated transparency discovery for journalists, auditors, and civil society
- Sectors: media, NGOs, academia
- Tools/workflows: AI agents that retrieve and summarize company disclosures, followed by human review; issue trackers mapping gaps to indicators
- Assumptions/dependencies: Human oversight remains necessary; reliability of agent retrieval; access to public content
- Investor and lender ESG-style assessment of AI governance
- Sectors: finance, venture capital, insurance
- Tools/workflows: FMTI-score lenses for investment committees; covenants requiring minimum indicator compliance; insurance underwriting criteria tied to post-deployment monitoring and mitigations
- Assumptions/dependencies: Accepted linkage between transparency and risk; portfolio monitoring; alignment with fiduciary duties
Long-Term Applications
- Regulatory baselines and certification programs using the FMTI indicator set
- Sectors: public policy, standards bodies
- Tools/workflows: codified disclosures (e.g., top-5 data sources, retention policies, reproducible risk evaluations) in regulation; third-party certification/audits; tiered compliance for general-purpose models
- Assumptions/dependencies: Legislative adoption; harmonization across jurisdictions; accredited audit ecosystem
- Machine-readable transparency schemas and disclosure APIs
- Sectors: software, compliance tech
- Tools/workflows: standardized JSON schemas for indicators; discoverable “/.well-known/ai-transparency” endpoints; automated compliance tooling that ingests and validates disclosures
- Assumptions/dependencies: Industry consensus on schemas; versioning; security controls for sensitive fields
- National infrastructure planning informed by compute/energy transparency
- Sectors: energy, utilities, planning agencies
- Tools/workflows: public reporting pipelines from AI developers and clouds; demand forecasting models for data centers; allocation policies balancing AI growth with grid/water constraints
- Assumptions/dependencies: Accurate metering; cooperation from private firms; integration with utility planning processes
- Train–test overlap auditing services and corpora registries
- Sectors: ML auditing, research
- Tools/workflows: third-party services that verify overlap against known corpora; registries of dataset hashes, provenance histories; contractual access to confidential training data summaries
- Assumptions/dependencies: Legal mechanisms to share data summaries/hashes; community registries; standardized overlap metrics
- Data licensing marketplaces with creator remuneration transparency
- Sectors: media, creator economy, legal tech
- Tools/workflows: standardized contracts disclosing rates, attribution, opt-out/opt-in; clearinghouses tracking compensation; provenance tokens recorded for licensed content ingestion
- Assumptions/dependencies: Platform participation; copyright settlement frameworks; viable monetization models
- Independent pre-deployment risk evaluation centers
- Sectors: safety labs, academia, government
- Tools/workflows: accredited labs with reproducible capability/risk batteries; secured access to pre-release models; publication of standardized risk summaries and mitigation adequacy assessments
- Assumptions/dependencies: Safe access pathways; funding; consensus on evaluation taxonomies
- Synthetic data governance and provenance tracking
- Sectors: ML platforms, compliance
- Tools/workflows: lineage tracking for human- vs. synthetic-generated data; quality and bias audits; disclosure of synthetic generation policies and instructions to data laborers
- Assumptions/dependencies: Tooling to trace data flows; standards for synthetic provenance; clarity on IP/privacy implications
- Model theft prevention technology stacks
- Sectors: cybersecurity, cloud
- Tools/workflows: robust watermarking of artifacts; anomaly detection for model exfiltration; disclosure and testing of theft mitigations (aligned with indicators)
- Assumptions/dependencies: Effective technical controls; interop with cloud security tooling; red-team validation
- Impact assessment frameworks for benefits and harms
- Sectors: public policy, economics, social science
- Tools/workflows: standardized measurement of economic/productivity gains and social risks by sector; longitudinal dashboards; guidance on high-risk deployment domains
- Assumptions/dependencies: Data access; methodological consensus; cooperation from deployers
- Consumer-facing transparency labels for AI products
- Sectors: consumer software, retail, education
- Tools/workflows: “AI Transparency Label” summarizing data sources, MBP, AUP enforcement, version history; browser/app store requirements to display labels
- Assumptions/dependencies: Usability and comprehension; platform policies; avoidance of dark-patterns
- Automated compliance agents for continuous transparency monitoring
- Sectors: compliance tech, DevOps
- Tools/workflows: agents that watch releases, docs, APIs to update transparency status; alerting on regressions (e.g., score drops); integration with governance boards
- Assumptions/dependencies: Stable data sources; robust verification; human-in-the-loop oversight
- Government procurement thresholds and incentives tied to transparency
- Sectors: public sector, defense, education
- Tools/workflows: minimum indicator compliance to bid; incentives (tax credits, fast-track certifications) for high transparency; public scorecards for suppliers
- Assumptions/dependencies: Policy adoption; enforcement capacity; balancing national security and transparency needs
- Interoperable agent protocol conformance programs
- Sectors: software agents, robotics
- Tools/workflows: test suites and badges indicating protocol support; marketplaces favoring interoperable agents; reduced vendor lock-in
- Assumptions/dependencies: Protocol standardization; industry adoption; governance of evolutions
These applications leverage the paper’s expanded indicator set (e.g., data acquisition, usage data, post-deployment monitoring, release practices), its methodological emphasis on reproducibility and taxonomies (capabilities, risks, mitigations), and its findings on where transparency deficits persist (training data, compute, usage/impact). Each depends on a mix of corporate incentives, legal constraints, standardization, and the maturation of supporting tooling and governance ecosystems.
Glossary
- Acceptable Use Policy (AUP): A formal set of rules governing permitted, restricted, and prohibited uses of an AI system and how enforcement is carried out. "Acceptable Use Policy (5 indicators)."
- Agent protocols: Standardized interfaces or specifications that enable AI agents to interoperate with tools, platforms, and other agents. "whether the developer discloses the supported agent protocols"
- AI bug bounty: A program that rewards external researchers for responsibly discovering and reporting vulnerabilities, misuse vectors, or risky behaviors in AI systems. "AI bug bounty, responsible disclosure policy, safe harbor"
- B2B: Business-to-business; companies that sell products or services to other businesses rather than directly to consumers. "enterprise-focused B2B companies"
- Compute allocation: The policies and practices determining how computational resources (e.g., GPUs, FLOPs) are distributed across training stages, experiments, and projects. "and how compute is allocated."
- Data provenance: Documentation of the origins, sources, and acquisition pathways of data used to train AI models. "Data acquisition and provenance remains poorly understood"
- Data retention policy: Rules specifying how long user or usage data is stored, under what conditions, and when it is deleted. "data retention policy"
- Distillation: A training technique in which a smaller or student model learns from the outputs or behaviors of a larger teacher model. "teacher model used for distillation"
- Downstream: The post-release part of the AI supply chain encompassing distribution, usage, and societal/economic impacts. "35 downstream indicators address the distribution and usage of models."
- Downstream Mitigations: Risk controls implemented in deployments and applications that use the model (e.g., detection of machine-generated content, enterprise-specific safeguards). "Downstream Mitigations (5 indicators)."
- EU AI Act: The European Union’s comprehensive regulation governing the development and deployment of AI systems, including transparency requirements for general-purpose models. "signatories to the EU AI Act General Purpose-AI Code of Practice."
- FLOPs: Floating point operations; a measure of computational work used to quantify training or inference cost. "( FLOPs, $570,000 - 760,000$ kWh, $2-300$ tCO2eq)"
- Flagship models: A developer’s premier, most prominent AI models that represent the company’s leading capabilities. "post-deployment usage and impact of their flagship models."
- Frontier Model Forum: An industry consortium of leading AI companies focused on advancing safety and responsibility in frontier AI systems. "The five members of the Frontier Model Forum we score end up in the middle of the Index"
- General Purpose-AI Code of Practice: A set of voluntary commitments under the EU AI Act aimed at guiding transparency and responsible practices for general-purpose AI developers. "EU AI Act General Purpose-AI Code of Practice."
- Model Behavior Policy (MBP): Developer-defined constraints, guidelines, and defaults that specify acceptable and unacceptable model outputs and responses. "Model Behavior Policy (4 indicators)."
- Model dependencies: External components (e.g., other models, tools) that a model relies upon during training, inference, or distillation. "model dependencies (\eg teacher model used for distillation)."
- Model theft: Unauthorized exfiltration, replication, or appropriation of a model’s weights, artifacts, or proprietary capabilities. "Adds an indicator on mitigations for model-theft."
- Open Source Initiative (OSI): The organization that defines criteria for open-source software and, in this context, clarifies what constitutes open-source AI artifacts. "the OSI definition of open-source AI"
- Open-weights: A release modality where model parameter weights are publicly available, enabling local inference, fine-tuning, and replication. "whether the developer provides open-weights access"
- Post-deployment Monitoring: Ongoing processes for tracking usage, detecting incidents, and mitigating risks after a model is released. "Post-deployment Monitoring (7 indicators)."
- Quantization: The process of reducing numerical precision of model parameters or activations (e.g., from float32 to int8) to improve efficiency with minimal performance loss. "quantization, terms-of-service"
- Responsible disclosure policy: A formal policy that defines how external researchers can report security or safety issues and how the developer will handle such reports. "AI bug bounty, responsible disclosure policy, safe harbor"
- Safe harbor: Legal or policy protections that shield external evaluators from liability when testing and reporting issues under defined conditions. "AI bug bounty, responsible disclosure policy, safe harbor"
- Synthetic data: Artificially generated data produced by models or algorithms rather than collected from real-world sources or human labor. "novel human-generated data, and synthetic data."
- Test-time scaling: Techniques that increase performance by allocating more computation during inference (e.g., longer reasoning chains, more sampling). "the technical paradigm with test-time scaling"
- Train-test overlap: The degree to which evaluation datasets contain content seen during training, which can inflate reported performance and undermine validity. "Adds a new indicator on train-test overlap."
- Upstream: The pre-release portion of the AI supply chain involving data acquisition, processing, compute, methods, and organizational resources. "32 upstream indicators address the resources involved in foundation model development."
- Versioning protocol: Formal procedures for naming, tracking, and documenting model versions and updates over time. "versioning protocol, change-log"
- Whistleblower protection: Organizational safeguards that protect employees who report misconduct, safety issues, or policy violations. "whistleblower protection policies"
Collections
Sign up for free to add this paper to one or more collections.