Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop

Abstract: The rapid advancement of LLMs has led to growing interest in using synthetic data to train future models. However, this creates a self-consuming retraining loop, where models are trained on their own outputs and may cause performance drops and induce emerging biases. In real-world applications, previously deployed LLMs may influence the data they generate, leading to a dynamic system driven by user feedback. For example, if a model continues to underserve users from a group, less query data will be collected from this particular demographic of users. In this study, we introduce the concept of \textbf{S}elf-\textbf{C}onsuming \textbf{P}erformative \textbf{L}oop (\textbf{SCPL}) and investigate the role of synthetic data in shaping bias during these dynamic iterative training processes under controlled performative feedback. This controlled setting is motivated by the inaccessibility of real-world user preference data from dynamic production systems, and enables us to isolate and analyze feedback-driven bias evolution in a principled manner. We focus on two types of loops, including the typical retraining setting and the incremental fine-tuning setting, which is largely underexplored. Through experiments on three real-world tasks, we find that the performative loop increases preference bias and decreases disparate bias. We design a reward-based rejection sampling strategy to mitigate the bias, moving towards more trustworthy self-improving systems.

• The paper demonstrates that iterative synthetic data retraining significantly amplifies bias, especially under incremental fine-tuning.
• It quantifies preference and disparate bias across news, creative, and math tasks using robust metrics like ROUGE-L, Bertscore, and Pass@1.
• It proposes a reward-guided sampling and reweighting strategy that effectively mitigates bias while maintaining generation quality.

Observations and Remedies for LLM Bias in Self-Consuming Performative Loop

Problem Formulation: Bias Dynamics in Self-Consuming Performative Loops

The recursive use of synthetic data for LLM retraining has become an increasingly prevalent paradigm, driven by the scarcity and contamination of pure human-generated datasets. This process yields a self-consuming retraining loop wherein each subsequent model generation ($\mathcal{M}_t$) is fine-tuned on synthetic samples from its predecessor ($\mathcal{M}_{t-1}$). The performativity of such systems—where user feedback and model performance mutually reinforce each other—further compounds the issue. Specifically, groups that are consistently underserved by an LLM contribute progressively less data, creating shifting distributions governed by implicit feedback mechanisms.

Figure 1: Illustration of the self-consuming performative loop for LLMs. Dynamic user feedback alters both synthetic data generation and retraining, driving bias evolution.

The paper formally introduces the Self-Consuming Performative Loop (SCPL), characterizing both standard retraining and incremental fine-tuning settings. In SCPL, group dynamics are explicitly modeled, typically distinguishing an advantaged group ($D^a$) and a disadvantaged group ($D^d$), with feedback affecting subsequent data ratios. Two practical data cycles are considered:

• Full Synthetic: Exclusive retraining on newly generated data per iteration.
• Accumulation: Accumulation-based retraining by concatenating previous synthetic and original datasets.

Bias Measurement and Experimental Setup

To quantitatively characterize bias, the study investigates three task domains: news continuation (political bias), preference dissection (creative vs. non-creative attributes), and math problem solving (easy vs. hard). For each, preference and disparate bias metrics are defined:

• Preference Bias: The fraction of model outputs classified as favoring the advantaged label, measured on unbiased test sets.
• Disparate Bias: The group-wise accuracy gap for tasks with ordinal difficulty (e.g., hard vs. easy math problems).

Figure 2: Preference bias and generation quality evolution using Qwen2.5-1.5B across News Continuation and Preference Dissection tasks.

For math reasoning, similarity metrics (ROUGE-L + Bertscore) capture the overall generation quality. The evaluation consistently applies robust classifiers and scoring models to track these phenomena across generations.

Figure 3: Disparate bias and Pass@1 accuracy for Qwen2.5-Math-7B on varying difficulty splits, highlighting bias evolution under SCPL.

Figure 4: Disparate bias and similarity evolution for Qwen2.5-Math-1.5B, demonstrating model scale sensitivity in bias dynamics.

Empirical Findings: Bias Amplification and Performance Degradation

The primary empirical observation is that iterative training with synthetic data in SCPL amplifies preference bias, especially under incremental fine-tuning regimes. SCPL exacerbates bias far more rapidly than non-dynamic self-consuming loops, where the data distribution remains fixed. The retraining setup yields only marginal bias amplification compared to incremental fine-tuning, which facilities compounding bias due to both model drift and biased data generation.

Figure 5: Preference bias and generation quality for Llama2-7B on the News task under incremental fine-tuning.

Incremental fine-tuning under performative feedback leads to pronounced increases in preference bias scores, notably on the News Continuation task with right-leaning articles. Degradation in generation quality is observed across all setups, with synthetic-only cycles consistently yielding the most rapid decay.

Accumulation partially mitigates the rate of bias amplification and generation quality decline but fails to revert models toward unbiased outputs, confirming results in prior literature.

Figure 6: Comparative bias mitigation performance of different data curation strategies using Qwen2.5-1.5B.

Model Scale and Data Ratio Sensitivity

Disparate bias, particularly in mathematical reasoning tasks, tends to decrease over SCPL iterations—indicating convergence in group performance but coupled with overall accuracy degradation. Notably, the rate of bias and performance change exhibits high sensitivity to initial group ratios, model scale, and the proportion of self-generated synthetic data. Smaller models (e.g., Qwen2.5-Math-1.5B) experience slower performance and bias decay under dynamic sampling, while larger models (Qwen2.5-Math-7B) display heightened sensitivity to distributional shifts.

Figure 7: Evolution of preference bias and generation quality for Llama2-7B in the incremental fine-tuning loop.

Figure 8: Variations in bias score and generation quality across News task retraining using Llama2-7B, showing greater stability in retraining versus fine-tuning.

Figure 9: Disparate bias and math problem solving ability trajectory under SCPL for Qwen2.5-Math-7B.

Bias Mitigation: Reward-Guided Sampling and Reweighting

To counteract bias amplification, the paper introduces a modular reward-based sampling procedure. For every candidate prompt, $k$ responses are sampled, scored by a reward function incorporating generation quality, group preference alignment, and task-specific criteria. Sample selection is then guided via a reweighting scheme that can preferentially over-sample from underrepresented (disadvantaged) groups. This strategy is shown to outperform naive rejection sampling and top-$k$ strategies in both bias reduction and preservation of generation quality across tasks.

Figure 10: Generation quality under multiple mitigation methods in the News task, highlighting the robustness of reward-reweighting.

Empirical results show that this approach is effective without requiring ground-truth labels or explicit bias evaluation during training, relying instead on proxy metrics and rule-based reward signals. The mitigation method is extensible, supporting additional properties such as sentiment, style, or output format.

Implications and Future Research Directions

These findings have several practical and theoretical implications:

• Deployment Risk: The use of synthetic data, especially under dynamic user feedback (SCPL), can rapidly concentrate model behavior toward dominant group preferences, amplifying social or systemic biases in deployed LLMs.
• Model Stability: Incremental fine-tuning, favored in real-world settings for data licensing or efficiency reasons, is particularly susceptible to compounding bias, and thus requires careful monitoring and curation.
• Mitigation Generalization: Reward-guided sampling offers a scalable and plug-and-play solution for bias reduction, adaptable to different tasks and domains. However, effectiveness depends on the fidelity of the reward model and the proxy metrics chosen.
• Theoretical Gaps: The contrast between bias reduction in DPO-style iterative training and bias amplification in SFT-based SCPL suggests deeper investigation into feedback dynamics and alignment methods is needed.

Future research should extend SCPL analysis to reinforcement learning protocols, multi-source and multi-model feedback loops, and integrate sophisticated preference modeling. Further investigation into group ratio, model scale, and domain transfer effects will be required for robust bias control in practical LLM systems.

Conclusion

This work provides a principled analysis of bias evolution in self-consuming performative loops for LLMs, establishing that dynamic feedback mechanisms and synthetic data reuse can dramatically amplify preference bias and degrade group fairness, especially under incremental fine-tuning. The proposed reward-based sampling and reweighting offer effective mitigation against these trends. The study calls for careful bias monitoring and dynamic data curation in future self-improving LLM deployments, and opens a fertile avenue for further research into performative feedback, multi-model ecosystems, and preference-aware training strategies.