Metacognitive Bias in Human and AI Systems

• Metacognitive bias is the systematic deviation between subjective confidence and actual performance, manifesting as overconfidence or underconfidence.
• Empirical studies show humans can exhibit significant bias on challenging tasks (up to +0.74) while LLMs typically display milder deviations.
• Resource-rational models and architectural adjustments, including explicit metacognitive prompts and feedback loops, provide practical approaches to mitigate bias.

Metacognitive bias denotes systematic deviations in confidence judgments relative to actual performance or correctness. It arises both in humans and artificial agents when beliefs about one’s own accuracy are misaligned, typically manifesting as overconfidence (confidence exceeding true accuracy) or underconfidence (the converse). In contemporary cognitive science and AI, metacognitive bias is a central focus for quantifying and correcting self-assessment errors, with significant consequences for individual decision-making, human-AI collaboration, and reliability of LLMs.

1. Formal Characterization and Quantitative Metrics

Metacognitive bias is formally captured as a signed mean error in confidence judgments. In operational terms, it is defined as the mean subjective confidence minus the actual proportion correct. Explicitly:

$\text{Bias} = \overline{C} - P_{\text{correct}}$

where $\overline{C}$ is the average confidence (on $[0,1]$ scale) and $P_{\text{correct}}$ is the fraction of correct responses (Pavlovic et al., 2024).

Related constructs include:

• Metacognitive sensitivity: Ability to discriminate between correct and incorrect responses, quantified by Sensitivity $$= \text{High-Confidence~Hit~Rate} - \text{High-Confidence~False~Alarm~Rate}$$.
• Calibration (Brier Score): Mean squared deviation between predicted confidence and outcome,

$$\text{Brier Score} = \frac{1}{N}\sum_{i=1}^N (p_i - o_i)^2$$

where $p_i$ is predicted probability, $o_i$ the outcome (Pavlovic et al., 2024).

In human-AI contexts, metacognitive bias is often measured by the average difference between users’ estimated and actual performance, $$\Delta EP_i = \text{Estimate}_i - \text{Performance}_i$$ (Fernandes et al., 2024).

2. Empirical Patterns in Humans and LLMs

Quantitative studies have established distinct bias profiles for both humans and LLMs. In comparative tests:

• Humans: Exhibit near-perfect calibration (bias $\approx$ 0) on simple items; marked overconfidence (bias $= +0.74$) on difficult or ambiguous items, consistent with the Dunning–Kruger phenomenon.
• LLMs: Generally display less metacognitive bias. On “best” items, models trend slightly underconfident (bias $\approx -0.05$ to $-0.10$). On “worst” items, mean overconfidence is lower ($+0.21$), with certain models (e.g., Llama 3, Mistral Large) even showing underconfidence (Pavlovic et al., 2024).

Human-AI interaction studies show that AI assistance increases user performance but also produces a consistent overestimation of ability (mean bias = +3.52 items on a 20-item test), with higher technical AI literacy correlating with greater metacognitive bias (Fernandes et al., 2024). Notably, the typical positive correlation between actual performance and overconfidence—diagnostic of Dunning–Kruger—vanishes under AI assistance, producing a uniform bias across ability groups.

3. Mechanistic Accounts: Resource Rationality and Algorithmic Factors

Resource-rational analysis provides a computational explanation for metacognitive bias, especially in humans. Nobandegani et al. propose a strategy selection model:

• Agents minimize mean squared error in outcome estimation using

$$\hat{E} = \frac{1}{\sum_{i=1}^s w_i} \sum_{i=1}^s w_i u(o_i), \quad o_i \sim q(o), \quad w_i = \frac{p(o_i)}{q(o_i)}$$

• The optimal sampling distribution for time-limited agents over-represents extreme outcomes due to the “metacognitive rationality factor” (MCRF)—formally, $$q^*_{meta}(o) \propto p(o) |u(o) - \mathbb{E}_p[u(o)]| \,\, \text{MCRF}(o, s)$$—explaining availability and framing biases as optimal under resource bounds (Nobandegani et al., 2018).

In LLMs, metacognitive bias emerges from architectural and training process deficits. The “metacognitive myopia” framework identifies five core mechanisms:

• Integration of invalid tokens/embeddings due to indiscriminant training,
• Repetition bias (probabilities inflated by token frequency),
• Conditioning failures (base-rate neglect),
• Frequency-based decision rules,
• Inappropriate higher-order inference (e.g., Simpson’s paradox) (Scholten et al., 2024).

4. Debiasing Interventions and Metacognitive Scaffolds

Several strategies have demonstrated efficacy in reducing metacognitive bias, both at the model and human-AI system level.

In LLM Architectures

• Explicit metacognitive monitoring and control modules assess source validity, redundancy, and factual calibration at generation time. If quality scores fall below threshold, generative control is activated: low-trust tokens are masked or external retrieval triggered, dynamically adjusting next-token sampling (Scholten et al., 2024).
• Meta-learning and human feedback loops iteratively calibrate downstream probabilities, especially in ambiguous contexts (Pavlovic et al., 2024).

Prompt Engineering

• The inclusion of explicit metacognitive natural-language cues such as “Could you be wrong?” prompts LLMs to produce extended outputs incorporating self-critique, error identification, counter-evidence, and bias awareness (Hills, 14 Jul 2025).
• Quantitatively, such meta-prompts reduce implicit bias (e.g., stereotypical pairings drop from 72% to 10%), eliminate metacognitive failures (from 96.3% to 0%), and increase reflection depth (number of distinct self-critiques per response) (Hills, 14 Jul 2025).

Human-AI Interaction Design

• Interfaces that insert deliberate “friction points”—forcing users to reflect on assumptions during prompt formulation and to scrutinize output with bias visualization overlays—can surface and attenuate anchoring and confirmation biases (Lim, 23 Apr 2025).
• Adaptive scaffolding mechanisms are being developed to adjust the frequency and complexity of interventions in accordance with user engagement and bias prevalence, though robust quantitative adaptation algorithms are still emerging (Lim, 23 Apr 2025).
• Confidence-calibration prompts and cognitive-forcing functions break automatic acceptance of AI suggestions, mitigating overreliance and restoring metacognitive monitoring (Fernandes et al., 2024).

5. Broader Theoretical and Practical Implications

Empirical dissociation between metacognitive sensitivity and bias challenges the notion that metacognitive bias is inseparable from conscious experience; algorithmic models and LLMs can exhibit near-human sensitivity with reduced bias (Pavlovic et al., 2024). The resource-rational account reframes availability and framing effects as optimal meta-level adaptations under computational limits, rather than cognitive “irrationality” (Nobandegani et al., 2018). In both human and artificial systems, interventions that elicit self-evaluation or inject counterfactual generation (“consider-the-opposite” strategies) consistently diminish bias and promote accuracy (Hills, 14 Jul 2025).

Overreliance on generative AI without metacognitive design not only increases individual bias but also risks institutionalizing overconfidence at scale, with implications for education, governance, and high-stakes decision workflows (Fernandes et al., 2024, Lim, 23 Apr 2025). Integrating reflective, adaptive metacognitive modules into both user-facing and system architectures is essential for trustworthy, bias-resilient AI-augmented cognition.

6. Illustrative Tables and Key Results

Metacognitive Bias Statistics in Humans and LLMs (Pavlovic et al., 2024):

Item Type	Humans	LLMs (Mean)	LLM Model Range
Best Responses	0.00	–0.07	–0.05 to –0.10
Worst Responses	+0.74	+0.21	–0.23 to +0.64

Meta-Prompting Effects (Hills, 14 Jul 2025):

Task Domain	Baseline Metric	Meta Prompt Metric	ΔBias / ΔFailure
Discriminatory Bias (B)	0.72	0.10	–0.62
Metacognitive Failures	96.3%	0.0%	–96.3 pp
Evidence Omission (E)	0	1	+1

Symptoms of Metacognitive Myopia in LLMs (Scholten et al., 2024):

Symptom	Origin	Downstream Effect
Integration of Invalid Embeddings	Indiscriminate token use	Persistent hallucinations
Repetition Bias	Token frequency in corpus	Majority opinions reinforced
Base Rate Neglect	Prompt conditioning	Overconfident unsupported outputs
Frequency-based Rules	Co-occurrence statistics	Status-quo bias, novelty suppression
Higher-Order Inference Failures	Aggregation blindness	Paradoxical or misleading conclusions

These results underscore the multi-level character of metacognitive bias—inherent to both computational strategy selection under constraint and the architectural regularities of large-scale statistical learners. Ongoing development of adaptive, reflective mechanisms in both LLMs and human-AI interfaces is central to mitigating systemic overconfidence, improving calibration, and aligning cognitive augmentation with robust self-monitoring.