Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions

Abstract: As LLMs transition from chat interfaces to integral components of stochastic pipelines across domains like educational assessment and synthetic data construction, the ability to faithfully sample from specified probability distributions has become a functional requirement rather than a theoretical curiosity. We present the first large-scale, statistically powered audit of native probabilistic sampling in frontier LLMs, benchmarking 11 models across 15 distributions. To disentangle failure modes, we employ a dual-protocol design: Batch Generation, where a model produces N=1000 samples within one response, and Independent Requests, comprising $N=1000$ stateless calls. We observe a sharp protocol asymmetry: batch generation achieves only modest statistical validity, with a 13% median pass rate, while independent requests collapse almost entirely, with 10 of 11 models passing none of the distributions. Beyond this asymmetry, we reveal that sampling fidelity degrades monotonically with distributional complexity and aggravates as the requested sampling horizon N increases. Finally, we demonstrate the propagation of these failures into downstream tasks: models fail to enforce uniform answer-position constraints in MCQ generation and systematically violate demographic targets in attribute-constrained text-to-image prompt synthesis. These findings indicate that current LLMs lack a functional internal sampler, necessitating the use of external tools for applications requiring statistical guarantees.

• The paper demonstrates that LLMs struggle to generate valid i.i.d. samples, with median pass rates around 13% in batch protocols.
• It shows that increased distributional complexity and larger sample sizes lead to poorer fidelity, evidenced by higher Wasserstein distances and failing KS tests.
• The study highlights downstream failures in applications like MCQ generation and text-to-image prompts, necessitating reliable external randomization tools.

LLMs and Probabilistic Sampling: An Empirical Audit of Distributional Fidelity

Motivation and Problem Definition

The paper "LLMs Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions" (2601.05414) presents a statistically grounded assessment of native probabilistic sampling capabilities in frontier LLMs. As LLMs are increasingly deployed as stochastic components in synthetic data pipelines, educational assessment, and generative agents, their ability to sample from user-specified probability distributions becomes a critical functional requirement, not merely a theoretical consideration.

The study addresses whether contemporary LLMs, without external tool augmentation, can generate i.i.d. samples from well-specified 1D target distributions with statistical validity at scale. The results challenge prevailing assumptions about LLMs' internal stochasticity, revealing systematic and protocol-dependent failures that have far-reaching implications for applied pipelines and model design.

Experimental Design and Methodology

A comprehensive evaluation pipeline benchmarks 11 state-of-the-art LLMs across a taxonomy of 15 probability distributions, stratified into three complexity tiers based on entropy, support characteristics, and tail properties. The experiment employs a dual-protocol design to disentangle distinct failure modes:

• Protocol A (Batch Generation): The model is prompted once to generate $N=1000$ samples in a single response, with samples produced sequentially in a shared context window.
• Protocol B (Independent Requests): The model receives $N=1000$ stateless calls, each requesting one sample, isolating its intrinsic distributional priors from contextual dependencies.

Distributional fidelity is quantified using rigorous statistical tests (two-sample Kolmogorov-Smirnov and Chi-square), Wasserstein-1 geometric distance ($\mathcal{W}_1$), and KL divergence against numpy.random reference samples. The pipeline also incorporates downstream assessments, including MCQ (multiple-choice question) answer position randomization and attribute-constrained text-to-image prompt synthesis.

Figure 1: Overview of the evaluation pipeline—11 LLMs are benchmarked across 15 distributions via batch and independent sampling; distributional fidelity is assessed with KS, $\chi^2$, and $\mathcal{W}_1$ metrics.

Main Empirical Findings

Protocol-Dependent Distributional Failures

The evaluation exposes a strong protocol-dependent asymmetry:

• In batch mode, statistical validity is modest at best: the median pass rate across distributions is 13%, with the top model at 40%.
• Under independent requests, performance collapses: 10 of 11 models fail to pass statistical validity on any distribution.

Batch generation’s apparent sampling capability is in fact emergent from autoregressive context—removal of context exposes the absence of a genuine internal sampler. This effect was pronounced even for Tier I distributions (Uniform, Gaussian), with $\mathcal{W}_1$ distances increasing by over an order of magnitude between batch and independent protocols.

Effect of Distributional Complexity

Sampling fidelity degrades monotonically with increasing distributional complexity. For Tier I (fundamental priors), select models can approximate valid samples in batch mode. For Tier III (heavy-tailed, complex, or multi-parameter), no LLM achieves valid sampling by any metric. Empirical $\mathcal{W}_1$ distance from the target distribution grows systematically with tier.

Figure 2: As distribution complexity increases, pass rates decrease and mean Wasserstein distance $\mathcal{W}_1$ rises across models, indicating declining fidelity.

Scaling with Sample Size

Conventional stochastic samplers exhibit improved fidelity with increasing $N$ via standard $\mathcal{O}(N^{-1/2})$ convergence. In contrast, both batch and independent protocols show inverse scaling: as $N$ increases, the Wasserstein distance and other error metrics worsen, and KS p-values collapse below significance thresholds. This reveals that statistical discrepancies are masked at small $N$ but unambiguously exposed as sampling horizons grow.

Figure 3: Increasing sample size $N$ leads to worsening KS p-values and $\mathcal{W}_1$ for DeepSeek-V3.2, highlighting non-convergent sampling behavior.

Downstream Application Failures

• MCQ Generation: Despite explicit instruction to uniformly randomize answer positions, all six tested models produce severe, statistically significant positional bias (e.g., correct answers appearing up to 54.6% in a particular position vs. 25% target).
• Attribute-Constrained Prompt Generation: When sampling demographic attributes and continuous values (e.g., height, coat color) for text-to-image prompts, models systematically violate the prescribed distributions—mode collapse, skewed demographics, and variance shrinkage persist even with precise numeric constraints in the prompt.

Theoretical Implications and Interpretability

The observed deficiencies demonstrate that current LLMs do not encode a functional i.i.d. sampler for probabilistic generation. Apparent randomness in LLM output is superficial and collapses when context is ablated. Structural limitations are amplified with increasing distributional complexity and sample size, which cannot be addressed merely by tuning decoding hyperparameters or prompt engineering. The result is contradictions to standard probabilistic convergence behaviors that underlie statistical simulation and fair data generation.

Practical Consequences

The failures documented here have immediate ramifications: synthetic data or randomized assignments generated without external tool use will not satisfy statistical validity requirements, potentially embedding systematic bias—especially dangerous in high-stakes domains such as educational assessment and fairness-constrained generative pipelines.

Where rigorous stochastic guarantees are required, LLMs must be instrumented to call trusted external samplers (e.g., numpy.random), rather than relying on native generation. Reliance on code generation or tool use is therefore not a convenience, but a necessity given present model architectures and training strategies.

Future Directions

This work motivates further research across several axes:

• Exploring architectural or objective modifications that explicitly endow models with the capacity for stochastic simulation and i.i.d. sampling.
• Developing hybrid LLM-tool frameworks that robustly integrate LLM reasoning with trusted algorithmic samplers for probabilistic tasks.
• Investigating whether fundamentally new pretraining corpora or data augmentation can promote internalization of probability-theoretic concepts—especially for rare, tail, or mathematically rich distributions.
• Systematically auditing downstream applications (e.g., fair data generation, randomized trial assignment, synthetic agents) for sampling infidelity and devising statistical safeguards.

Conclusion

This systematic benchmark establishes that current-generation LLMs do not possess a reliable internal mechanism for probabilistic sampling across a broad spectrum of distributions and practical applications. Statistical validity is context- and protocol-dependent, degrades with complexity and sample size, and is not preserved under decomposition into stateless calls. For any application demanding distributional guarantees, claims of LLM stochasticity must be treated with skepticism, and external randomization tools remain indispensable for reliable, unbiased generation.

(2601.05414)