Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology

Abstract: LLMs perform strongly on biological benchmarks, raising concerns that they may help novice actors acquire dual-use laboratory skills. Yet, whether this translates to improved human performance in the physical laboratory remains unclear. To address this, we conducted a pre-registered, investigator-blinded, randomized controlled trial (June-August 2025; n = 153) evaluating whether LLMs improve novice performance in tasks that collectively model a viral reverse genetics workflow. We observed no significant difference in the primary endpoint of workflow completion (5.2% LLM vs. 6.6% Internet; P = 0.759), nor in the success rate of individual tasks. However, the LLM arm had numerically higher success rates in four of the five tasks, most notably for the cell culture task (68.8% LLM vs. 55.3% Internet; P = 0.059). Post-hoc Bayesian modeling of pooled data estimates an approximate 1.4-fold increase (95% CrI 0.74-2.62) in success for a "typical" reverse genetics task under LLM assistance. Ordinal regression modelling suggests that participants in the LLM arm were more likely to progress through intermediate steps across all tasks (posterior probability of a positive effect: 81%-96%). Overall, mid-2025 LLMs did not substantially increase novice completion of complex laboratory procedures but were associated with a modest performance benefit. These results reveal a gap between in silico benchmarks and real-world utility, underscoring the need for physical-world validation of AI biosecurity assessments as model capabilities and user proficiency evolve.

• The paper demonstrates that mid-2025 LLMs moderately enhance procedural progression in novice biology labs but do not significantly increase full protocol completion.
• Using Bayesian modeling and an RCT design, the study quantifies a risk ratio of 1.42, highlighting modest performance uplifts in tasks like cell culture.
• Findings reveal that while LLM engagement speeds up task initiation and intermediate steps, overall benefits are limited by novice elicitation challenges.

Measuring LLM-Assisted Novice Performance in Physical Biology Laboratories

Experimental Design and Cohort Characterization

The study executed an investigator-blinded, two-arm randomized controlled trial (RCT) evaluating the effect of mid-2025 frontier LLMs on the acquisition of dual-use laboratory skills by novice participants—specifically within the context of a viral reverse genetics workflow. The trial enrolled 153 participants over eight weeks, with randomization between an Internet-only control arm and an LLM-assisted intervention arm. Both arms worked independently in a BSL-2 laboratory across five tasks: micropipetting, cell culture, molecular cloning, virus production, and RNA quantification. Notable is the deliberate absence of detailed protocols or guidance—participants were provided only high-level objectives.

Participants were predominantly undergraduate students with minimal prior hands-on biology exposure, balanced across baseline characteristics (Figure 1).

Figure 1: Baseline participant demographics, illustrating education level, field, and prior biology experience.

Primary Outcomes: LLM-Assisted Laboratory Execution

The pre-registered primary outcome was completion of the “core reverse genetics sequence” (Tasks 2–4: cell culture, molecular cloning, virus production). Completion rates were extremely low, with no significant difference between arms (LLM: 5.2%; Internet: 6.6%; $P = 0.759$, RR = 0.79, 95% CI 0.24–2.62), indicating that current LLM models do not materially increase the likelihood of novice completion of complex biological workflows.

Secondary, per-task analyses revealed nonsignificant trends favoring LLMs, especially for cell culture (Task 2: LLM 68.8% vs. Internet 55.3%, $P = 0.059$). However, the effect was significant in the per-protocol cohort (LLM: 79.7% vs. Internet: 62.5%; $P = 0.025$, RR = 1.28), demonstrating that, under ideal adherence, LLMs moderately increased success in less tacit-intensive tasks.

Bayesian Pooled Modeling: Quantification of Average Uplift

A hierarchical Bayesian logistic regression, partially pooling across tasks and participants, estimated a risk ratio for LLM assistance of $\sim$1.4-fold (posterior mean RR = 1.42, 95% CrI 0.74–2.62, $Pr(RR > 1) = 85.5\%$). Sensitivity analyses across model specifications demonstrated robustness of this finding (Figure 2).

Figure 2: Posterior risk ratios from Bayesian pooled models indicate modest expected LLM-frequency uplift in reverse genetics tasks.

This empirically grounds the claim that mid-2025 LLMs confer moderate performance benefit, while strongly excluding high-magnitude (e.g., $>2.6\times$) increases in novice laboratory capability for reverse genetics workflows.

Procedural Progression: Acceleration in Protocol Advancement

Analysis of procedural stepwise advancement, via Bayesian ordinal regression, revealed that LLM participants were more likely to advance to intermediate stages in almost all tasks (odds ratio range: 1.2–1.4, posterior probability of positive effect: 81%–96%). LLM participants initiated tasks more frequently and overcame early procedural bottlenecks at higher rates (see Sankey diagrams in Figure 3 for cell culture).

Figure 3: Stepwise completion for cell culture (Task 2), showing LLM arm participants progressing further through milestones.

Additionally, LLM-assisted participants succeeded at cell culture significantly faster (mean RMST reduction: $-6.0$ days, $P = 0.02$) and with fewer attempts, confirming efficiency benefits in protocol execution.

Usage Patterns and Elicitation Constraints

LLM arm participants demonstrated substantial engagement (23 prompts/day; 60% with image uploads; heavy frontier model usage), though usage intensity did not predict outcome. Notably, YouTube was cited as more useful than any individual LLM, indicating the continued importance of demonstrative, tacit knowledge transfer via video rather than text-based model interaction. Time to first reagent request in molecular cloning was faster for LLM arm, but time to correct request was comparable (Figure 4), illustrating difficulty in eliciting specialized LLM capabilities for sequence analysis and reagent selection.

Figure 4: Time to first reagent request, showing LLM arm's faster initiation but comparable correctness.

Perception surveys demonstrated that LLM participants' confidence in LLM helpfulness decreased over time, while Internet-only participants’ beliefs about hypothetical LLM benefit increased, reflecting practical elicitation and model limitations rather than differential task difficulty.

Implications for Benchmarks, Safety, and Evaluation Methodology

Findings highlight a significant gap between in silico LLM benchmark performance and real-world novice utility in physically executed tacit-rich laboratory tasks. Expert-dependent benchmarks often overestimate LLM-mediated uplift, as expert users are able to critically incorporate model outputs and novice users lack the procedural context and prompting sophistication. This underlines the necessity for evaluation frameworks contextualized by both user expertise and physical task environment.

These results constrain policy-relevant estimates of AI biosecurity risk: empirical 95% bounds rule out high-magnitude uplift in novice reverse genetics capability for mid-2025 LLMs. Moreover, stepwise protocol metrics expose latent LLM benefits missed by binary success measures, providing methodological guidance for future "human uplift" studies in biosecurity-critical domains.

Practical uplift from LLMs in synthetic biology is currently modest—limited by model accuracy on specialized tasks, interface suitability for tacit knowledge, and users' ability to elicit correct outputs. Ongoing improvements in AI model biological specialization (e.g., Biomni Lab), user interface innovation (AR-guided demonstration), and participant proficiency will likely modulate results in future deployments.

Limitations and Future Directions

The study’s low completion rates, even in the Internet arm, suggest the protocols posed unexpectedly high difficulty for novices; power was insufficient to detect small treatment effects due to optimistic prior effect size forecasts. Tasks were decoupled and simplified, limiting direct inference to real-world, end-to-end viral synthesis. Rapid evolution in LLM capabilities and user expertise further restricts the scope of future generalization.

A path forward is the continuous, empirical, and context-aware reassessment of human-AI interaction, incorporating developments in biology-specific models, user interfaces, and AI-facilitated laboratory automation. Such adaptive evaluation is crucial for evidence-based biosecurity policy and risk governance.

Conclusion

This large, tightly controlled human study demonstrates that access to mid-2025 frontier LLMs provides modest but non-negligible uplift for novices attempting complex, dual-use biology laboratory protocols. While LLMs facilitate procedural initiation and accelerate progress through workflow steps, they do not substantially increase ultimate completion rates. There is clear divergence from digital benchmark projections, underscoring the requirement for physical-world validation in AI biosecurity risk assessments. Stepwise procedural metrics and Bayesian modeling approaches offer rigorous methodological advances for ongoing measurement of AI-enabled human performance in laboratory contexts.