Data Visualization Literacy Assessment

Updated 1 February 2026

Data Visualization Literacy Assessment is the empirical measurement of how accurately individuals interpret visual representations using metrics like P-HIC and Rasch modeling.
It integrates predictive modeling and statistical analysis, using features such as item difficulty, human performance, and demographic data to forecast interpretation correctness.
The methodology supports adaptive assessments and personalized training by calibrating item difficulty and refining evaluation through rigorous experimental design.

Data visualization literacy assessment refers to the empirical measurement and prediction of an individual's ability to correctly interpret, extract information from, and reason about data visualizations. The assessment of this competence is grounded in quantitative metrics evaluating Human (or Population) Interpretation Correctness (P-HIC), psychometric modeling of item difficulty, experimental design for response collection, and predictive modeling to anticipate or adapt to heterogeneity in user interpretation. The following sections synthesize current state-of-the-art methodologies, metrics, findings, and future directions for rigorous assessment of data visualization literacy.

1. Formalization of Human Interpretation Correctness (P-HIC)

The foundational metric in data visualization literacy assessment is Human Interpretation Correctness (P-HIC), generally defined as the empirical probability, or model-based estimate, that a human user will interpret a data visualization (or item) correctly under controlled conditions. In the most basic operational form, for a visualization item $i$ and a user response $y_i \in \{0,1\}$ (0: correct, 1: incorrect), P-HIC is specified as a Bernoulli probability:

$\text{P-HIC}(x) = \Pr(Y = 0 \mid x)$

where $x$ is a vector of features capturing item difficulty, human profile characteristics, and prior performance (Falessi et al., 28 Jan 2026).

For assessment scenarios involving explanations (e.g., saliency maps, prototype-based rationales), P-HIC measures the fraction of human subjects who can correctly infer the model’s predicted class or output given the same information supplied to the model (Davoodi et al., 2023). This extends naturally to binary correctness in annotation workflows (Long et al., 13 Aug 2025) and subjective semantic coding (Chochlakis et al., 22 May 2025).

Across domains, the precise operationalization of P-HIC is task-specific but always resolves to a statistically grounded, objective criterion for correctness as defined by ground-truth or consensus model output.

2. Predictive Modeling of Interpretation Correctness

The latest advances in data visualization literacy emphasize not just retrospective scoring (did the user get it right/explanation match?), but prospective modeling: predicting, prior to exposure, whether a given person will likely interpret a specific visualization correctly or not. This is operationalized as a supervised binary classification task (Falessi et al., 28 Jan 2026).

The features used span:

Item Difficulty Metrics:
- RaschDifficulty: logit-scale difficulty parameter $\beta_i$ from Rasch item-response models.
- ExpertDifficulty: median of discrete expert ratings for each visualization item.
Human Profile: Self-reported demographics (e.g., age, gender, country, education, native language, expertise, years of experience), one-hot encoded.
Human Performance:
- PercCorrect: Running fraction of prior items answered correctly.
- MedianDifficulty: Median RaschDifficulty of previous items, contextualizing performance by underlying task complexity.

The logistic regression model with feature selection outperforms random forests and multilayer perceptrons for the prediction of item-level correctness, with a median AUC of 0.724 and median Cohen’s kappa of 0.319 over 32 datasets. RaschDifficulty is the dominant feature, supporting the centrality of psychometric modeling in visualization literacy assessment (Falessi et al., 28 Jan 2026).

3. Assessment Protocols and Experimental Design

Data visualization literacy assessment is typically conducted via large-scale online surveys or controlled experiments:

Participants and Items: For general literacy, datasets include over a thousand respondents and hundreds of visualization items spanning types such as Name, Function, and Content questions (Falessi et al., 28 Jan 2026).
Response Collection: Each subject answers a block of items derived from randomized orderings of core visualization types, ensuring session-level statistics can be modeled (e.g., adaptation over time).
Gold-Standard and Distractors: Tasks present a target question (e.g., “what is the most common value depicted?”), with the correct choice plus plausible distractors.
Expert Calibration: Item difficulty can be established by expert ratings and refined via human response modeling (Rasch).
Cross-validation: Ten-time ten-fold cross-validation or similar robust evaluation regimes support generalizability estimation.

In explanation/interpretability studies, designs involve explicit manipulation of information provided to the user (explanation/no explanation), forced-choice multiple-choice responses (e.g., “which label did the model predict?”) (Shen et al., 2020), and controlled presentation of explanations (e.g., saliency maps, prototypes) (Davoodi et al., 2023, Kim et al., 2021).

4. Quantitative Metrics and Analysis

Assessment leverages several complementary metrics to measure P-HIC and its correlates:

Metric	Mathematical Expression / Description	Primary Reference
Percent Correct	$\text{P-HIC} = \frac{\# \text{correct}}{N}$	(Shen et al., 2020, Davoodi et al., 2023, Falessi et al., 28 Jan 2026)
RaschDifficulty	$\beta_i$ in $P(X_{ni}=1) = \frac{e^{\theta_n - \beta_i}}{1+e^{\theta_n - \beta_i}}$	(Falessi et al., 28 Jan 2026)
Cohen’s Kappa	$\kappa = \frac{p_o - p_e}{1 - p_e}$	(Long et al., 13 Aug 2025, Falessi et al., 28 Jan 2026)
AUC (ROC)	$\mathrm{AUC} = \int^1_0 \mathrm{TPR}(t) d\mathrm{FPR}(t)$	(Falessi et al., 28 Jan 2026)
Thematic Similarity	$0$ (dissimilar), $0.5$ (partial), $1.0$ (nearly identical)	(Long et al., 13 Aug 2025)
Attention Correctness	$AC(y_t) = \sum_{i\in R_t} \hat{\alpha}_{ti}$	(Liu et al., 2016)
Balanced Accuracy	$\text{P-HIC}_{\text{bal}} = \tfrac{1}{2}(\text{TPR} + [1-\text{FPR}])$	(Kim et al., 2021)

Statistical hypothesis testing (unpaired or paired $t$ -tests, significance relative to chance) is standard (Shen et al., 2020, Kim et al., 2021). Researchers also analyze feature importance (e.g., Gain Ratio) and conduct ablation studies to understand the marginal impact of each predictor (Falessi et al., 28 Jan 2026).

5. Human Interpretation: Explanations, Error Types, and Item Heterogeneity

Experimental results show marked heterogeneity in interpretation correctness:

Post-hoc explanations can degrade P-HIC: Saliency map visual explanations decreased average accuracy by 10%—p=0.01—in label-guessing tasks, with the largest effect for errors due to similar appearance or context-based correlations (Shen et al., 2020).
Prototype-based models show strong variation: ProtoPNet and TesNet yield $>92\%$ human prediction correctness, while ProtoPool collapses to $<50\%$ due to many-to-many prototype-class assignments (Davoodi et al., 2023).
Human interpretation accuracy is highest for concrete, less ambiguous items (e.g., “title,” “aims”), with $\kappa$ and thematic similarity $>0.90$ ; lowest for interpretive or underspecified questions (e.g., theoretical frameworks), with $\kappa$ sometimes $\approx0.25$ (Long et al., 13 Aug 2025).
Variations in performance result from item-specific difficulty, explanation method, interpretive ambiguity, and individual background. RaschDifficulty is consistently the dominant predictor across sessions and item types (Falessi et al., 28 Jan 2026).

6. Psychometrics and Adaptive Assessment

Rich psychometric modeling is foundational for pinpointing and predicting human interpretation of visualizations:

Rasch Modeling: Estimation of item-specific difficulties ( $\beta_i$ ) and user abilities ( $\theta_n$ ) on the logit scale, supporting principled item selection and performance profiling (Falessi et al., 28 Jan 2026).
Adaptive Item Selection: P-HIC-based prediction allows dynamic item selection, reducing participant burden and discouragement by avoiding items predicted to be “too difficult” for a given user.
Personalized Training: Monitoring user PercCorrect relative to item RaschDifficulty enables just-in-time delivery of targeted practice, optimizing learning curves and calibration (Falessi et al., 28 Jan 2026).
Assessment Calibration: Repeated extractions (including AI-AI “self-consistency” runs) reveal item ambiguity or interpretive complexity, guiding refinement of instruments and coding manuals (Long et al., 13 Aug 2025).

7. Limitations, Best Practices, and Future Directions

Current data visualization literacy assessment is robust but must account for several practical and methodological constraints:

Item Calibration: Accurate and stable RaschDifficulty and expert ratings are critical; leave-one-out estimation strategies must be used to avoid data leakage (Falessi et al., 28 Jan 2026).
Limited Demographic Prediction: Human Profile features add little predictive power beyond psychometric and performance signals (Falessi et al., 28 Jan 2026).
Explanation Fidelity: Methods assigning multiple classes to the same explanation primitive (e.g., prototypes) reduce interpretability and should be limited (Davoodi et al., 2023).
Evaluation Rigor: Standard practice calls for reporting the full confusion matrix rather than marginal accuracy; significance against chance has to be established via appropriate hypothesis testing (Kim et al., 2021).
Task-Dependence: Correctness varies steeply with the specificity and interpretive richness of the task. Categorical and literal extraction yields high P-HIC; subjective or abstract questions yield low consistency regardless of whether the extractor is human or AI (Long et al., 13 Aug 2025).

Areas for further research include multidimensional modeling of P-HIC (incorporating confidence, reasoning alignment, and temporal consistency), rich integration with AI-generated interpretations and correction protocols, and direct measurement of interpretive diversity as a desirable attribute rather than mere noise (Long et al., 13 Aug 2025, Chochlakis et al., 22 May 2025).

In sum, data visualization literacy assessment now synthesizes psychometric modeling, rigorous statistical analysis, and predictive modeling to anticipate and optimize human interpretation correctness. This methodological integration supports both robust measurement and practical applications in adaptive assessment and personalized visualization training (Falessi et al., 28 Jan 2026, Davoodi et al., 2023, Kim et al., 2021, Long et al., 13 Aug 2025, Liu et al., 2016, Shen et al., 2020).