Human-AI Collaboration in Science at Scale: A Global Large-scale Randomized Field Experiment

Abstract: Collaboration is the defining mode of modern science, yet its core mechanism -- feedback -- remains hard to observe, difficult to scale, and unequally distributed. Here we test whether LLMs can contribute to this hidden but vital practice and reallocate scientific feedback, an essential yet scarce resource for knowledge production. In a global large-scale randomized field experiment, we delivered customized LLM-generated feedback for over 31,000 arXiv preprints across 150 fields and more than 45,000 researchers from 133 geographic regions. Relative to controls, authors who received feedback had a significantly higher likelihood of revising their manuscripts, corresponding to a 12.55% relative increase over the baseline revision rate. Exposure to AI feedback also increased authors' subsequent use of LLM tools in their future papers, suggesting longer-run shifts in scientific practice. These effects were strongest among authors from non-English-dominant research regions, manuscripts less embedded in the scholarly literature, and teams with lower h-indexes and earlier career stages, consistent with the idea that AI feedback may provide the greatest benefit where access to timely critique is otherwise limited. Together, these findings provide causal evidence that structured AI-based interventions can transform access to scientific feedback from a largely private advantage into a more widely distributed resource, with broader implications for productivity, equity, and capacity across the global research system.

• The paper demonstrates that AI-generated feedback increases manuscript revisions by 12.55%, serving as a collaborative equalizer in science.
• It employs a stratified randomization design on over 31,000 arXiv preprints, ensuring robust causal inference through controlled intervention.
• Findings show a 5.29% rise in long-term LLM adoption, with significant benefits for marginalized, low-resource scientific communities.

Human-AI Collaboration in Scientific Feedback: Randomized Evidence at Global Scale

Experiment Overview and Motivation

The paper "Human-AI Collaboration in Science at Scale: A Global Large-scale Randomized Field Experiment" (2605.24180) presents a rigorous, randomized study assessing the impact of LLM-generated feedback on scientific practice. By targeting over 31,000 arXiv preprints and more than 45,000 authors across 133 regions and 150 scientific fields, the study addresses critical questions about the scalability and equity of collaborative feedback—a core mechanism underpinning modern scientific progress.

Traditional scientific feedback is a scarce and unequally distributed resource, often stratified by geographic, linguistic, and institutional boundaries. The intervention introduces controlled, structured AI-generated critique during a pivotal phase of LLM adoption, allowing for robust causal inference about both immediate and persistent behavioral changes in scientific practice.

Methodology

The experiment adopts a stratified randomization protocol, assigning first-version arXiv preprints either to a treatment group (receiving customized LLM feedback via secure links) or a control group (receiving no intervention). The LLM system leverages multi-agent pipelines and in-context learning to provide tailored review comments, suggested title improvements, and grammar checks.

Short-term outcome measures focus on revision activity (number of version updates within one month post-feedback), while long-term effects track the subsequent adoption of LLM tools using state-of-the-art AI detection models applied to follow-up publications. The design mitigates confounding factors inherent to observational AI productivity studies, such as endogenous tool adoption and user heterogeneity, by exogenously administering standardized feedback.

Key Results

Short-Term Effects

Authors in the treatment arm showed a statistically significant 12.55% increase in manuscript revision rate relative to controls. Exploratory analysis indicates these revisions are directionally more substantive (conceptual, ethics, novelty) rather than restricted to linguistic or typographic corrections.

Heterogeneity and Equity Effects

Treatment effects were most pronounced among:

• Non-English-dominant regions: 19.9% relative increase in revision rate
• Low scholarly embeddedness: 26.4% relative increase
• Lower author h-index: 20.2% increase
• Earlier career stage: 18.9% increase

In contrast, high-resource (English-dominant, highly cited, senior) groups showed no statistically significant effect. This pattern strongly supports the claim that AI-generated feedback functions as a collaborative equalizer, disproportionately benefitting traditionally marginalized or under-networked constituencies in science.

Long-Term Adoption Effects

Among authors with minimal prior LLM usage, receipt of AI feedback led to a 5.29% relative increase in measurable LLM adoption in subsequent publications over twelve months. Equitable—and statistically significant—patterns of adoption mirrored those found in revision behaviors, with heightened effects among low-resource groups. This suggests not merely transient experimentation but persistent integration of AI into research workflows.

Practical and Theoretical Implications

The findings provide robust causal evidence that AI, especially LLMs, can transform access to scientific feedback from an exclusive, private advantage to a distributed, scalable resource, augmenting both productivity and equity. The study demonstrates feasibility for global-scale, institutionally randomized interventions targeting live scientific manuscripts, a methodological advance for empirical science of science research.

From a practical standpoint, AI feedback interventions could lower barriers to professional advancement among early-career or non-native English scholars, accelerate iterative improvement cycles in research, and catalyze broader tool adoption.

Theoretically, the experiment circumvents key confounds in prior productivity studies by exogenous assignment and protocol-standardized critique. The observed equalizing effect initiates empirical contributions to ongoing debates about AI’s role in remediating persistent inequalities in scientific knowledge production.

Risks and Future Directions

The potential risks identified include the possibility of homogenization as scientists begin to rely on similar LLM models for feedback, which could contract the spectrum of critique and reduce collective methodological diversity. As AI feedback becomes more ubiquitous, preserving conceptual and disciplinary heterogeneity will require deliberate architectural and institutional countermeasures.

Future research directions will pivot to multi-model architectures, varying critique protocols, and incentive structures rewarding dissent and diverse methodological perspectives. As baseline LLM usage rises and further integration occurs, experimental frameworks like this will be critical for benchmarking and understanding evolving collaborative dynamics in an AI-infused scientific system.

Conclusion

This randomized field experiment provides compelling evidence that AI-generated feedback can prompt substantive scientific revisions, drive sustained adoption of LLM tools, and most critically function as a collaborative equalizer—benefitting authors and teams outside traditional feedback-rich networks. The study advances both practical deployment strategies for AI-human collaborative systems and theoretical accounts of how AI can reshape the collaborative fabric of scientific practice, with important implications for productivity, equity, and the global distribution of scientific opportunity.