Selective Test-Time Debiasing for CLIP via Reward Gating

Abstract: Vision LLMs (VLMs) demonstrate strong zero-shot performance, but often perpetuate social stereotypes in person-centric queries, yielding skewed demographic distributions. Current debiasing methods apply uniform bias corrections across all input queries regardless of their bias sensitivity, creating a fundamental fairness--utility trade-off. Strong debiasing distorts semantically meaningful information in bias-insensitive queries, while weak debiasing fails to mitigate stereotypes in bias-sensitive ones. This one-size-fits-all approach hampers simultaneously achieving high utility on bias-insensitive queries and fairness on bias-sensitive queries. We introduce Reward-Gated Test-Time Adaptation (RG-TTA), a reinforcement learning-based test-time adaptation framework that selectively applies debiasing based on input sensitivity. RG-TTA adaptively triggers fairness regularization based on the bias sensitivity of each input during test-time policy adaptation, while focusing exclusively on optimizing cross-modal alignment for bias-insensitive inputs. Experiments on fairness benchmarks (e.g., FairFace, UTKFace) demonstrate substantial bias reduction while simultaneously improving zero-shot utility, resolving the trade-off of uniform debiasing.

• The paper introduces a reward-gated test-time adaptation approach that selectively applies fairness regularization for bias-sensitive queries.
• It measures semantic–attribute alignment to decide when to trigger debiasing, balancing mitigation of social bias with maintaining utility.
• Experimental results demonstrate significant reductions in demographic skew alongside improved zero-shot accuracy with manageable runtime overhead.

Selective Test-Time Debiasing for CLIP via Reward Gating

Problem Formulation and Motivation

This paper addresses the persistent issue of social bias in vision-LLMs (VLMs), specifically in CLIP-like architectures, where models trained on large-scale web data frequently internalize societal stereotypes. These biases become manifest in person-centric queries, resulting in skewed distributions across protected demographic attributes (e.g., gender, age, race), with notable failures such as stereotypical associations of occupations or underrepresentation of minoritized groups. Existing debiasing methodologies conventionally apply bias mitigation uniformly across all queries, disregarding query-specific sensitivity to bias. This monolithic approach enforces a strict fairness–utility trade-off: aggressive debiasing degrades zero-shot performance on bias-insensitive queries, while conservative debiasing leaves biases largely unmitigated in sensitive scenarios.

Reward-Gated Test-Time Adaptation (RG-TTA)

The paper proposes Reward-Gated Test-Time Adaptation (RG-TTA), an RL-based inference-time intervention that implements selective debiasing. The core innovation is a per-query reward-gating mechanism: for each input, the model first assesses bias sensitivity by measuring the semantic–attribute alignment. If the query is determined to be bias-sensitive (i.e., a high degree of demographic entanglement is detected via a closeness test to a reference attribute distribution), the fairness-regularized objective is activated; otherwise, the model strictly optimizes cross-modal alignment.

The approach updates only the query-modality encoder (text or image, as appropriate), enforcing parameter resets at the conclusion of each episode to avoid catastrophic interference across queries.

Selective Gating Strategy

The gating function $\delta(q)$ is defined as an indicator based on a discrepancy heuristic comparing CLIP alignment between the top-1 match of $q$ and the mean over a protected attribute reference set. When the semantic–attribute distance falls below a threshold $\epsilon$, the fairness regularizer is triggered. This yields conditional test-time adaptation: debiasing is invoked only when warranted by the input’s demographic sensitivity.

Adaptive Reward Formulation

The reward for test-time parameter updates combines two terms:

• CLIP Alignment Reward: Maintains overall image–text matching performance, calculated as a baseline-corrected alignment score from a reference CLIP model.
• Attribute-Balancing Regularizer: When debiasing is gated on, a projection onto a precomputed bias subspace (obtained via PCA over class prototypes in protected attribute space) provides a reward that encourages a more uniform distribution over the attribute classes, penalizing over-represented groups and favoring under-represented ones.

The final per-candidate reward is a weighted sum, with the balance determined by $\delta(q)$.

Optimization Protocol

For each query, the model runs a small number of policy-gradient updates on a truncated candidate set (with $K$ candidates, chosen adaptively). After each query, parameters are reset to prevent update-induced drift. This episodic adaptation is directly tractable at inference due to test-time efficiency considerations.

Experimental Results

Extensive evaluation on in-domain (UTKFace, FairFace) and out-of-domain (FACET) fairness benchmarks demonstrates that RG-TTA consistently reduces demographic skew (as measured by MaxSkew and NDKL), often outperforming static test-time debiasing baselines such as CLIP-clip, Biased-prompts, and Joint V-L. Crucially, this bias reduction does not come at the expense of utility: ImageNet zero-shot top-1 accuracy increases relative to the original model and baseline debiasing methods. The composite ABLE metric, which harmonizes fairness and accuracy, shows substantial improvement as well.

For example, when evaluated on gender and age (UTKFace, FairFace, FACET), RG-TTA achieves:

• Gender MaxSkew (e.g., on UTKFace): reduced from 0.114 (original) to 0.051; ImageNet Top-1 accuracy improved from 68.31 to 70.38;
• Out-of-domain generalization: significant MaxSkew and NDKL reduction on FACET, with ImageNet/Flickr performance retained or improved.

Ablations confirm that selective gating is critical; globally applied debiasing yields strong fairness but severely degraded utility. The benefit holds across architecture scales (ViT-B/16/32, ViT-L/14) and reference model choices. Cross-dataset generalization, intersectional debiasing, and extension to non-demographic spurious bias (e.g., Waterbirds background) underscore the principled flexibility of the method.

Analysis and Observations

Gating Mechanism

Empirical analyses show that the gate’s false-positive and false-negative rates are negligible under a proxy labeling scheme: nearly all bias-sensitive queries are correctly flagged, and virtually no bias-insensitive queries are unnecessarily debiased. Threshold sensitivity experiments reveal that the main hyperparameter ($\epsilon$) is robust within a practical range. Failure cases are rare and typically involve complex query semantics where salient non-attribute cues confound the gate.

Runtime Considerations

RG-TTA introduces moderate overhead (relative to vanilla CLIP inference): $\sim$111 ms for bias-insensitive queries and up to $\sim$467 ms for bias-sensitive queries (ViT-B/16, A6000 GPU), due to per-query adaptation and candidate evaluation. In typical deployments, the majority of queries are bias-insensitive, making the overall inference latency acceptable for fairness-critical applications. Critically, the framework is training-free and requires no retraining or domain-specific fine-tuning, facilitating rapid adaptation to evolving fairness needs.

Attribute and Model Scalability

The reward-gated principle generalizes to intersectional attributes by re-defining the bias subspace over cross-products (e.g., gender × race). The approach is directly applicable to alternative VLMs (e.g., BLIP) with minor threshold recalibration. However, when cross-attribute class cardinality grows, the reliability of distributional estimation in the subspace may degrade.

Limitations and Future Directions

Notable limitations include runtime overhead on bias-sensitive queries compared to offline strategies; potential gate sensitivity to reference set and attribute estimator choices; and the dependency on a possibly larger reward model for alignment feedback. The current design focuses on single-attribute and moderate-dimensional intersectional fairness; scaling to higher-cardinality or multi-attribute debiasing requires more sophisticated reward compositionality and constraint handling. Robustness to reward signal bias, attribute estimator error, and extension to more challenging open-world settings are open research avenues.

Potential directions include efficient candidate selection schemes, scalable and compositional gating/reward mechanisms for complex attribute spaces, and plug-and-play adaptation strategies for emerging large multimodal architectures.

Conclusion

The paper establishes selective, input-dependent debiasing as a practical and effective paradigm for fairness in VLMs. By employing reward-gated test-time adaptation, the method resolves the fairness–utility dilemma endemic to uniform debiasing: bias is substantially reduced for sensitive queries, yet general zero-shot performance is preserved or improved on neutral inputs. The episodic, reward-driven protocol is model-agnostic, training-free, and amenable to rapid deployment scenarios, representing a compelling intervention for mitigating bias in contemporary and future multimodal AI systems.