Face-Only Counterfactual Evaluation Paradigm

• The paper introduces a counterfactual evaluation paradigm that isolates facial demographic effects using a two-stage image editing pipeline, evidenced by precise metrics (e.g., facial edit intensity differences).
• It employs a robust methodology combining face localization and attribute-driven counterfactual generation to maintain invariant non-facial context.
• Empirical evaluations on real and synthetic datasets reveal systematic biases in decisions such as salary inference, highlighting disparities across race–gender groups.

A face-only counterfactual evaluation paradigm is a principled framework for measuring the impact of facial demographic attributes—such as race and gender—on the behavior of vision-LLMs (VLMs) or face-understanding systems, under strict visual control. By generating image pairs (“counterfactual twins”) from real or synthetic data that differ exclusively in targeted facial traits while preserving all other visual factors, this paradigm enables rigorous, attributionally clean audits of model bias in scenarios where visual confounding is otherwise unavoidable (Chen et al., 11 Jan 2026, Ramesh et al., 2024).

1. Motivation and Problem Statement

In conventional benchmarking for social bias, real-world images entangle demographic features with confounding visual attributes: background, clothing, camera angle, lighting, and more. This confounding makes it difficult to ascribe model disparities in outcomes (e.g., biased salary inferences or occupation recognition) specifically to facial demography rather than unrelated visual context. Prior approaches trade off between realism (using in-the-wild photographs) and control (relying on synthetic, but potentially less realistic, images). The face-only counterfactual evaluation paradigm aims to bridge this gap, enabling isolation of the causal effect of face attributes by holding all non-facial content invariant. This paradigm can also be executed purely in the synthetic regime, where face images are generated or edited to tightly match attributes and preserve identity, as articulated in (Ramesh et al., 2024).

2. Technical Methodology

The heart of this paradigm lies in a two-stage image editing or synthesis pipeline:

Face Localization and Masking:

A facial region is first detected and isolated within a source image $I \in \mathbb{R}^{H \times W \times 3}$ using a face-landmark tool (e.g., MediaPipe). The face mask $M \in \{0,1\}^{H \times W}$ splits the image into facial content $I_{\text{face}}$ and context $I_{\text{ctx}}$, with $I = I_{\text{face}} + I_{\text{ctx}}$.

Attribute-Driven Counterfactual Generation:

A facial editor—a commercial diffusion-based model such as Google’s “Nano Banana Pro” for real photographs (Chen et al., 11 Jan 2026), or a latent-space diffusion model with semantic guidance (“SEGA”) for synthetic faces (Ramesh et al., 2024)—is applied to replace the facial content with a version reflecting target demographic attributes $\mathbf{a}_{\mathrm{target}}$, producing a counterfactual face. The edited image

$$I_{\mathrm{cf}} = (1-M) \odot I + M \odot G(I_{\text{face}}, \Delta \mathbf{a})$$

preserves all non-face pixels, ensuring that only controlled demographic variables are manipulated.

Synthetic Attribute Editing (SEGA):

For fully synthetic pipelines, a source face image $x$ is mapped to a counterfactual $x' = g_{a_i}(x)$ via guided diffusion, with explicit objectives for attribute correctness, specificity (no collateral attribute flips), and identity preservation (measured in an embedding space) (Ramesh et al., 2024). Semantic gradients based on CLIP-aligned text/image similarity steer the generation at each denoising step.

3. Datasets: FOCUS and Synthetic Counterfactual Sets

FOCUS (Face-Only Counterfactuals from Real Photos):

A dataset of 480 counterfactual portraits is constructed from six professional occupations, five race categories (White, Black, Asian, Latino, Middle Eastern), and two gender groupings. For each occupation, eight high-quality, neutral source photos are located; for each, ten counterfactuals varying only in facial race–gender attributes are synthesized. Audits confirm localization (mean facial edit intensity ≈ 0.14 vs non-face ≈ 0.02) and demographic accuracy (GPT-4o label match 97.9%) (Chen et al., 11 Jan 2026).

Synthetic Counterfactual Data (Editor’s term):

A large corpus is produced (e.g., 15,542 valid samples across 135/152 demographic–attribute cells), using generative pipelines based on Stable Diffusion fine-tuned on faces, semantic editing gradients, strict attribute verification and human audits. Edits encompass 19 binary facial attributes (accessories, age, hair style, facial hair, etc.) over 8 demographic strata (joint race–gender categories) (Ramesh et al., 2024).

Dataset	Data Type	Control Level	Scope	Paper
FOCUS	Real photos	Maximal (face-only)	480 images, 6 occupations, 10 demos.	(Chen et al., 11 Jan 2026)
Synthetic CF	Synthetic faces	Maximal (full face)	15,542 images, 8 demos., 19 attrs	(Ramesh et al., 2024)

4. Benchmarking and Evaluation Tasks

REFLECT Benchmark Suite (on FOCUS):

Three controlled, decision-oriented tasks probe VLM bias:

• Two-Alternative Forced Choice (2AFC):

Model chooses between two matched face variants (same photo, different race–gender face), under prompts referencing Income, Education, or Perceived Safety. WinRate metrics for each group indicate directional preference; deviation from 50% reveals bias.

• Multiple-Choice Questions (MCQ):

Models predict a salary bin (A–F) or education level (A–D) for a single image. Group-level gaps computed as $$\Delta_h = (\mu_h - \mu_{h_{\mathrm{ref}}})/\mu_{h_{\mathrm{ref}}}$$.

• Salary Recommendation:

Given a portrait plus occupation and biography, models assign a continuous salary amount. Demographic gaps computed relative to White (for races) or Female (for gender).

Synthetic Evaluation Protocols:

Synthetic counterfactual faces are processed by commercial computer vision systems (e.g., Instagram’s Image Understanding service), measuring average concept score shifts ($\Delta S_c(d, a_i)$) per demographic and attribute. Robustness is probed by how recognition concept scores (e.g., “face,” “beard,” “hair_long”) degrade or shift under targeted edits; fairness is measured by disparities in these shifts across demographics (Ramesh et al., 2024).

5. Key Empirical Findings

Persistence and Modulation of Demographic Bias:

VLMs (GPT-5, Gemini-2.5-Pro, Qwen-3-VL-Plus, DeepSeek-VL2, Llama-3.2-90B) evaluated using the face-only counterfactual paradigm on FOCUS demonstrate:

• Demographic disparities persist even when only the face changes. Income, education, and safety judgements reveal systematic preference for certain race–gender groups under strict context control.
• Task design shapes the magnitude and direction of bias: Income 2AFC prompts most strongly favor male, White faces; Perceived Safety reverses to female preference; education exhibits model-dependent patterns.
• Model-specific polarization: state-of-the-art VLMs display varying levels of separation; some models (e.g., GPT-5, Gemini) exhibit large win-rate and mean-gap disparities, while others (Llama, Qwen) show milder but still nontrivial effects (Chen et al., 11 Jan 2026).

Robustness and Fairness Under Synthetic Face Edits:

Classical vision systems evaluated with synthetic face-only counterfactuals show:

• Major semantic concept detection (e.g., “face”) degrades substantially under facial-accessory edits (e.g., facemasks), with penalties varying by demographic (e.g., facemask edit yields a −0.09 score drop for Black Male, vs. −0.22 for Asian Male).
• Attribute edits reveal group-dependent robustness failures and unfairness; for instance, certain attributes (sunglasses, beard, hair changes) disproportionately affect detection for specific groups (Ramesh et al., 2024).

6. Implications and Prospects

The face-only counterfactual evaluation paradigm redefines the standard for bias audits in multimodal and vision models by eliminating attributional ambiguity arising from background confounding. Its strict invariance—preserving all but the face—enables clean causal measurement of protected attribute effects, supporting both regulatory compliance and ethical scrutiny in high-stakes deployments (e.g., personnel decisions, law enforcement, financial screening) (Chen et al., 11 Jan 2026).

Potential Extensions:

• Incorporation of identity-preserving constraints in editing to mitigate within-face artifact drift.
• Expansion to other demographic axes such as age, disability, and voice accent cues.
• Scalable dataset generation covering more occupations, cultural contexts, and demographic groups.
• Integration of counterfactual derivation into debiasing loops or fine-tuning pipelines.

A plausible implication is that generalization and fairness improvements may require not only audit via these paradigms but also training-time interventions based on counterfactual data.

7. Related Methodological Frameworks

The face-only counterfactual paradigm spans both real-photo editing (Chen et al., 11 Jan 2026) and fully synthetic generation (Ramesh et al., 2024). Both approaches rely on guided generative models constrained for attribute specificity and identity preservation. The paradigm’s mathematical basis—edit operator $g_{a_i}$, transformation metric $\mu_h$, WinRate, and concept score shifts—sets a rigorous foundation for compositional, explainable model probing. The synthetic variant’s combination of diffusion-based editing, automatic artifact filtering (SVMs, GPT-4o), and human audits demonstrates scalability and extensibility for systematic robustness and fairness evaluation.

Key papers:

• "Measuring Social Bias in Vision-LLMs with Face-Only Counterfactuals from Real Photos" (Chen et al., 11 Jan 2026)
• "Synthetic Counterfactual Faces" (Ramesh et al., 2024)