LLMs Are Not a Silver Bullet: A Case Study on Software Fairness

Abstract: Fairness is a critical requirement for human-related, high-stakes software systems, motivating extensive research on bias mitigation. Prior work has largely focused on tabular data settings using traditional Machine Learning (ML) methods. With the rapid rise of LLMs, recent studies have begun to explore their use for bias mitigation in the same setting. However, it remains unclear whether LLM-based methods offer advantages over traditional ML methods, leaving software engineers without clear guidance for practical adoption. To address this gap, we present a large-scale study comparing state-of-the-art ML- and LLM-based bias mitigation methods. We find that ML-based methods consistently outperform LLM-based methods in both fairness and predictive performance, with even strong LLMs failing to surpass established ML baselines. To understand why prior LLM-based studies report favorable results, we analyze their evaluation settings and show that these gains are largely driven by artificially balanced test data rather than realistic imbalanced distributions. We further observe that existing LLM-based methods primarily rely on in-context learning and thus fail to leverage all available training data. Motivated by this, we explore supervised fine-tuning on the full training set and find that, while it achieves competitive results, its advantages over traditional ML methods remain limited. These findings suggest that LLMs are not a silver bullet for software fairness.

• The paper presents a rigorous comparison of traditional ML and LLM-based bias mitigation, showing ML methods reliably reduce fairness disparities (SPD, EOD, AOD) across tasks.
• The methodology uses a unified experimental protocol on real-world datasets (Adult, Compas, Credit) and evaluates multiple mitigation strategies across six fairness tasks.
• Key results indicate that while fine-tuning improves LLM performance, traditional ML pipelines remain more effective and cost-efficient for fairness-critical applications.

Large-Scale Empirical Analysis of ML and LLM-Based Methods for Software Fairness

Introduction

This paper conducts a comprehensive and systematic evaluation of bias mitigation strategies for fairness-critical tabular classification tasks, directly comparing state-of-the-art traditional ML methods and LLM-based paradigms. The investigation is motivated by the increasing interest in leveraging LLMs across software engineering tasks, including fairness-sensitive domains, and the unclear practical advantage of LLM-based methods over established ML approaches.

Experimental Framework and Evaluation Protocol

The empirical analysis encompasses three canonical real-world tabular datasets: Adult, Compas, and Credit, each covering distinct domains (income prediction, recidivism risk assessment, and credit scoring) and fairness-sensitive attributes (sex, race, age). Across six fairness evaluation tasks, the study benchmarks eight representative ML-based mitigation algorithms spanning pre-processing, in-processing, post-processing, and ensemble strategies, using four canonical ML classifiers (LR, RF, SVM, DNN). Parallelly, eight LLM-based methods—covering zero-shot, in-context learning with random and fairness-aware demonstration selection, label manipulation, and FCG variants—are evaluated with modern LLMs (primarily GPT-4o-mini, with followup experiments on Qwen3-Max-Thinking, GPT-5, Gemini-2.5-flash, and DeepSeek-v3.2 for model robustness).

A rigorous, unified experimental protocol ensures comparability. All experiments report macro performance (Precision, Recall, F1, Accuracy) and group fairness (SPD, EOD, AOD) metrics, with repeated runs for statistical significance. LLM-based tabular processing uses controlled prompt serialization:

Figure 1: Prompt structure for the LLM paradigm, encompassing task description, $k$-shot demonstrations, and a query instance.

Comparative Analysis: Traditional ML vs. LLM-Based Bias Mitigation

The empirical results establish a robust performance and fairness gap in favor of the traditional ML paradigm:

• Across all six tasks and all metrics, ML-based bias mitigation consistently yields lower SPD, EOD, and AOD (48.3%–59.6% reduction relative to LLMs), and stronger predictive performance (11.1% higher Accuracy, 9.3% higher F1, 11.3% higher Precision, and 4.6% higher Recall on average).
• These disparities are statistically significant in 72.2% of fairness and 83.3% of predictive tests.
• The gap prevails even with advanced, larger-scale or reasoning-enhanced LLMs (Qwen3-Max-Thinking, GPT-5, etc.), demonstrating the ML paradigm's resilience to architectural developments in LLMs.

Claim: LLMs, even with advanced architectures or larger context capabilities, do not outperform traditional ML in realistic tabular bias mitigation settings. Simply scaling or enhancing reasoning in LLMs does not eliminate this gap.

Sensitivity to Evaluation Setting

A critical finding is that fairness efficacy in LLM-based paradigms is highly sensitive to evaluation design:

• Methods evaluated using artificially balanced test distributions (common in prior LLM studies) report considerably improved fairness—SPD, EOD, and AOD are inflated by up to 138.4%, 49.7%, and 36.2% under natural, imbalanced test distributions.
• The apparent superiority of LLM-based approaches in some literature thus stems from discrepancies in experimental protocol, not from intrinsic algorithmic merit.

Training Paradigm: Demonstration-Limited ICL Versus Fine-Tuning

Traditional LLM-based approaches (in-context learning) leverage only a limited subset of the training data (16 demonstrations in $k$-shot prompts), in contrast to ML methods which utilize the entire dataset for model fitting. Fine-tuning LLMs on the entire training set:

• Substantially improves LLM-based model performance and fairness relative to ICL (SPD, EOD, and AOD decrease by 42.3%–54.0%; predictive metrics increase by 0.8%–15%).
• Augmenting fine-tuning with classic ML pre-processing (e.g., COT, LTDD, FairMask) further increases fairness.
• Despite these gains, fine-tuned LLMs only match or slightly exceed traditional ML methods in roughly half of metric/test combinations, with statistical significance in a minority of cases.
• The cost (compute, monetary) and practical complexity of LLM fine-tuning are non-trivial, often exceeding ML pipelines for modest or no improvement.

Implications for Fairness-Critical Software Engineering

This evidence set yields several actionable theoretical and practical implications:

• LLMs are not a universal solution for fairness-sensitive tabular tasks: Established ML mitigation remains the empirical and cost-effective optimum.
• Evaluation setting is decisive: Fairness claims based on balanced tests are not robust or generalizable to real-world, naturally imbalanced distributions.
• Cross-paradigm comparison is essential: Results confined to ML or LLM-only paradigms can be misleading; comprehensive benchmarks should be protocol- and paradigm-inclusive.
• Fine-tuning is necessary but insufficient: While access to all training data yields substantial improvement over prompt-limited ICL/zero-shot approaches, this does not translate into practical superiority over specialized ML bias mitigation pipelines.

Conclusion

The study provides a decisive statement regarding the current landscape of software fairness for tabular data: empirically, neither in-context learning nor supervised-finetuned LLM methods surpass the combination of targeted traditional ML bias mitigation and domain-appropriate classifiers. Theoretical advances in LLM capability and demonstration handling have not, so far, resulted in a practical paradigm shift for fairness assurance in tabular SE tasks. Practitioners and researchers should avoid the uncritical deployment of LLM-based approaches for fairness-critical domains without rigorous, cross-paradigm, and protocol-aligned evaluation.

Future Directions

Future research will need to revisit the data representation, supervision strategies, and architectural integration of LLMs with tabular data; isolate where current LLMs fall short (representation, bias removal, sample efficiency); and devise benchmarks that robustly capture real-world fairness-critical conditions. Advances in prompt retrieval, synthetic demonstration generation, hybrid neuro-symbolic models, or explicit optimization for fairness-regularized objectives may incrementally close the gap identified in this work.

---

Figure 1: Prompt structure for LLM-based tabular prediction, illustrating the sequence of task description, demonstrations, and test query used for in-context learning and all LLM-based evaluations.