WEIRD Metric: Quantifying Research Bias

• WEIRD Metric is a quantitative framework that defines research bias using five demographic dimensions at the country level.
• It employs specific national indicators and statistical methods like Kendall's τ to measure dataset representation and sampling imbalance.
• The framework supports transparency and guides improvements in social media studies by highlighting over-representation and encouraging inclusive data practices.

The WEIRD metric is a quantitative framework designed to assess the extent to which research—particularly in social computing—over-represents populations that are Western, Educated, Industrialized, Rich, and Democratic. Originally formulated to highlight limitations in psychological research, the WEIRD construct has been adapted to empirical analyses of social media studies, most systematically by Septiandri et al. in the context of the ICWSM conference (Septiandri et al., 2024). This operationalization enables rigorous, reproducible measurement of both the provenance and demographic characteristics of research datasets and supports transparent reporting and mitigation of sampling bias.

1. Formal Operationalization of WEIRD Dimensions

Each WEIRD dimension is defined at the country level, enabling computation of paper- and corpus-level metrics:

• Western (W): Binary indicator $W_n$, where $W_n=1$ if country $n$ is part of the Western group under Huntington’s Clash-of-Civilizations (EU members, U.S., Canada, Australia, New Zealand), $0$ otherwise.
• Educated (E): Mean years of schooling for adults $\geq 25$ years, $E_n$ (UNDP Human Development Report).
• Industrialized (I): Competitive Industrial Performance index, $I_n$ (UNIDO).
• Rich (R): Gross National Income per capita (PPP), $R_n$ (World Bank).
• Democratic (D): Political rights score, $D_n$ (Freedom House), with higher scores indicating greater freedom.

No composite WEIRD scalar is calculated; scores along each dimension are reported independently, permitting multidimensional profiling of sampling bias.

2. Mathematical Framework and Score Computation

The method for calculating representation and WEIRD scores is as follows:

1. Country-level dataset representation:

$s_n = \sum_{p=1}^{P} r_{p,n}$

where $r_{p,n}$ is the fraction of data in paper $p$ from country $n$. For $k$ datasets in a paper, each is weighted $1/k$. If no explicit country information is available, $r_{p,n}$ is set to the platform's penetration rate in country $n$.

2. Normalization (“paper-ratio” vector):

$$\psi_n = \frac{\left(s_n \,/\, \sum_m s_m\right)}{\left(\pi_n \,/\, \sum_m \pi_m\right)}$$

with $\pi_n$ the population of country $n$.

1. WEIRD dimension scores:
   • Western:

   $W = \frac{1}{N}\sum_{n=1}^{N} W_n$

• Educated:

  $E = \tau\left(\{\psi_n\},\,\{E_n\}\right)$

• Industrialized:

  $I = \tau\left(\{\psi_n\},\,\{I_n\}\right)$

• Rich:

  $R = \tau\left(\{\psi_n\},\,\{R_n\}\right)$

• Democratic:

  $D = \tau\left(\{\psi_n\},\,\{D_n\}\right)$

Here, $\tau$ is Kendall's rank correlation coefficient, ranging from –1 (over-focus on low-value countries) to +1 (over-focus on high-value countries); 0 indicates proportionate sampling.

3. Data Sourcing, Annotation, and Validation

The empirical basis comprises 494 ICWSM conference papers (2018–2022), filtered to exclude 74 papers lacking country information or relying solely on synthetic data, yielding a final corpus of 420. Data extraction was performed by 188 crowdworkers based in Anglophone Western countries, recruited via Prolific with stringent quality requirements (≥95% approval, ≥50 HITs). Annotations included author affiliations, dataset sources, and user distribution by country, using a custom HTML interface with embedded attention checks. Only 38.2% of submissions were error-free; thus, two authors manually corrected annotations for 216 papers for reliability.

4. Statistical Inference and Interpretation

WEIRD scores are interpreted at both the individual paper and corpus level. Papers are categorized as “exclusively Western” ($\forall n, W_n=1$), “exclusively non-Western” ($\forall n, W_n=0$), or “mixed.” For E, I, R, and D scores, Kendall’s $\tau$ is calculated with 95% confidence intervals, estimated by 10,000 bootstrap samples. Chi-squared tests examine associations between paper type (full, dataset, poster) and WEIRDness; permutation tests compare EIRD scores across paper classes. There is no absolute threshold for “highly WEIRD”; interpretation is comparative, particularly with reference to CHI and FAccT conferences.

5. Comparative Empirical Findings

Major quantitative findings are summarized in the table below:

Venue	Exclusively Western (%)	Exclusively non-Western (%)	Mixed (%)	E (Kendall τ)	I	R	D
ICWSM	37.4	12.1	50.5	0.36	0.35	0.49	0.32
FAccT	84.4	7.0	8.6	0.31	0.01	0.34	0.37
CHI	75.9	18.3	5.8	0.43	0.27	0.50	0.51

Confidence intervals for ICWSM: E [0.23, 0.49], I [0.20, 0.50], R [0.36, 0.61], D [0.20, 0.45].

Dataset and poster papers at ICWSM display lower E and D scores (Δτ ≈ 0.12 for E, 0.18 for D) than full papers. Cross-country authorship correlates with sampling less Democratic (ρ = –0.16) and less Educated (ρ = –0.11) populations. This suggests that collaboration across countries may partially counterbalance WEIRD bias, though not uniformly across all dimensions.

6. Recommendations for Mitigating WEIRD Bias

Concrete recommendations for research practice include:

• Expanding publication checklists to require explicit disclosure of:
  1. Countries represented in datasets.
  2. Social media platforms analyzed.
  3. Affiliation countries of authors.

With such information, paper-ratio and WEIRD scores can be computed automatically for transparency, not for gatekeeping.

• Mandating a “Responsible AI” or “Impact” statement to disclose demographic or political risks arising from dataset choices.
• Promoting author diversity through scholarships, ‘shadow mentoring,’ and multinational collaboration, especially for researchers from under-represented regions.

A stepwise “recipe” for quantifying WEIRD bias includes: (1) identifying user countries or inferring via penetration rates, (2) computing $r_{p,n}$, $s_n$, and $\psi_n$, (3) referencing global indicators for W, E, I, R, D, (4) calculating WEIRD scores, and (5) benchmarking against known conference statistics. A plausible implication is that such standardized reporting may advance transparency, comparability, and representation in computational social science datasets (Septiandri et al., 2024).

7. Applications and Broader Significance

The WEIRD metric provides a reproducible tool for evaluating the representativeness of social media research and can be directly applied to any study with sufficient country-level granularity. Its implementation in the ICWSM context demonstrates that social computing research, while exhibiting lower WEIRD bias than other venues, remains skewed toward Educated, Industrialized, and Rich populations. The public methodological checklist serves as both a diagnostic and a transparency tool, facilitating awareness and accountability concerning global sampling in research. This operationalization supports field-wide initiatives for research equity and responsible data practice, with immediate implications for reproducibility and policy adoption.