Relevance-Based Metrics

• Relevance-based metrics are quantitative measures that assess the topicality and appropriateness of system outputs based on semantic alignment.
• They employ diverse methodologies—including probabilistic models, causal inference, and deep embedding techniques—to evaluate dialogue, retrieval, and ML tasks.
• Recent advances focus on improving reference-free scoring, reducing domain sensitivity, and enhancing interpretability to better align with human judgments.

Relevance-based metrics quantify the "aboutness," topicality, or appropriateness of system outputs in dialogue, information retrieval, machine learning, and multi-modal tasks. Unlike surface-level measures (n-gram overlap, raw accuracy), these metrics aim to estimate how well a generated response, system prediction, or retrieval result aligns with the underlying information need, user context, or semantic intent. The concept has been formalized in probabilistic, information-theoretic, causal, knowledge-aware, and embedding-based frameworks, with recent advances emphasizing reference-free scoring, robustness to domain shifts, interpretability, and correlation with human judgments.

1. Foundations and Formal Definitions

Relevance metrics typically assess the degree to which a candidate output (response, prediction, document, caption) is on-topic or addresses the salient aspects of the preceding context, query, or information need. In open-domain dialogue, this corresponds to the logical continuity with respect to prior turns, penalizing generic or off-topic replies (Berlot-Attwell et al., 2022). In IR, relevance is operationalized as the utility, gain, or answerability of a result for a given query, often with graded scales or probabilistic models of assessor-user agreement (Demeester et al., 2015). In supervised ML tasks where multiple outcomes are plausible per context, metrics such as the Relevance Score reflect the empirical distribution of acceptable outputs rather than strict match (Gopalakrishna et al., 2013).

Information-theoretic relevance is formalized via mutual information and conditional dependence (Feng et al., 2024):

• For context $c$ and response $r$:

$$I(c_i; r) = E_{c_i, r} [ \log \frac{p(c_i, r)}{p(c_i)p(r)} ]$$

• Conditional mutual information:

$$I(c_i; r \mid c_j) = E_{c_i, c_j, r} [ \log \frac{p(c_i, r\mid c_j)}{p(c_i\mid c_j)p(r\mid c_j)} ]$$

Empirical metrics use classifier-based approximations and logistic regression over deep model features, for instance in the IDK metric for dialogue, which predicts $$\text{relevance}(c, r) = \sigma(w^\top x(c, r) + b)$$ using BERT NSP embeddings and sparse linear weights (Berlot-Attwell et al., 2022).

2. Methodological Taxonomy: Models and Algorithms

Relevance-based metrics deploy diverse computational strategies, often motivated by the limitations of reference-based or pure similarity methods:

• Logistic Regression over Deep Features (IDK): Fixed BERT NSP head, sparse logistic regression trained on positives (true next-turns) and a single negative example ("i don't know"), yielding rapid trainability and domain robustness (Berlot-Attwell et al., 2022).
• Causal Dependence Classification (CausalScore): Ensemble of unconditional and conditional RoBERTa-based classifiers to approximate mutual information between each context utterance and response. Aggregation over detected "cause" utterances produces a score in $[0,1]$ reflecting causal history-response ties (Feng et al., 2024).
• Augmented Reference Cosine Similarity (MARS): Reinforced self-planning LM infilling produces context-enriched reference sets, followed by weighted cosine similarity with candidate outputs. Masking, RL-guided infill, and dynamic references boost human alignment (Liu et al., 2021).
• Prompt-based Pairwise Relevance (MetricPrompt): Reformulates few-shot classification as binary MLM-style cloze relevance. Contextual prompt encoding, meta-verbalizer sum over synonym tokens, and mean-max pooling enable robust, label-agnostic classification (Dong et al., 2023).
• Disagreement-based Prediction (PRM): Probabilistically estimates the expected user relevance given observed assessor grades, then transforms IR metrics (Precision, nDCG) via gain functions with disagreement parameters $p_{R|\ell}$ (Demeester et al., 2015).
• Data-driven Continuous Relevance for Ranking (DCG_φ): Smooth Hermite interpolants map raw item scores onto $[0,1]$ per-query relevance, giving graded gains for position-discounted metrics (nDCG) and penalizing order errors proportionally to underlying score gaps (Moniz et al., 2016).
• Knowledge Gap-aware Scoring: Models background and target user knowledge vectors, computes gap-filling potential per document, and re-ranks or evaluates sessions by total "gap closure" per cost of interaction (Ghafourian, 2022).
• Concept-distance Matching (nn-IoU): For medical CBIR, calculates approximate concept intersection over union based on distances in medical knowledge graphs, allowing semantic matches below strict string equality (Wei et al., 16 Jun 2025).
• Layerwise Attribution Relevance (GAE): For neural explainability, aggregates faithfulness, robustness, and class contrastiveness via perturbation and attribution score alignment, producing a unified $[0,1]$ metric for ranked attribution methods (Vukadin et al., 2024).
• Multi-modal and Curriculum-based Metrics: Exam-style coverage and answerability ratings (EXAM), context-conditioned image/text relevance, and fusion of vision-language embeddings further extend relevance quantification beyond pure text or retrieval (Farzi et al., 2024, Sun et al., 2024, Jiang et al., 2019).

3. Empirical Validation and Domain Robustness

Recent studies reveal severe domain sensitivity in many relevance metrics, particularly those based on LM probabilities or fine-tuned embedding similarity. For instance, NUP-BERT relevance shows sensitivity ratios up to 11.6 across test domains (Berlot-Attwell et al., 2022). By contrast, IDK reduces this to ~3.9, achieving 37%-66% reduction in measured domain sensitivity and state-of-the-art human correlation on the HUMOD dataset (Spearman/Pearson $\rho, r=0.58$). CausalScore delivers Pearson $r\approx0.29$-$0.33$, Spearman $\rho\approx0.33$-$0.42$—doubling the best prior metrics (GPT-4: $r\approx0.16$) on diverse dialogue sets (Feng et al., 2024).

Across NLG, MARS outperforms seven unsupervised metrics (BLEU, METEOR, ROUGE, MoverScore, BERTScore) in all tested story, summary, and QA tasks, demonstrating high resistance ($\Delta r$ drops) to adversarial inputs such as token reordering (Liu et al., 2021).

Relevance-based metric strengths are further reflected in:

• High Spearman/Kendall leaderboard correlation in EXAM metrics ($\rho\approx0.94$, $\tau\approx0.84$) (Farzi et al., 2024).
• Improved accuracy in scoring model predictions that are empirically plausible but not strictly matching the observed outcome distributions, as evidenced by RS outperforming CA in lighting control evaluation (Gopalakrishna et al., 2013).
• Fine-grained image captioning evaluation, where REO (relevance, extraness, omission) outperforms composite metrics like SPICE in correlation with human preference and error disentanglement (Jiang et al., 2019).

4. Interpretability, Efficiency, and Practical Deployment

Relevance-based metrics increasingly emphasize transparency, modularity, and low annotation or computation cost:

• IDK leverages frozen BERT features, single negative example, and L₁-sparse regression, avoiding expensive fine-tuning; training can be completed in minutes and transfered cross-domain (Berlot-Attwell et al., 2022).
• CausalScore employs classifier-based conditional independence tests rather than direct MI estimation, coupled with self-training and domain adaptation via labeled/unlabeled dialogue logs (Feng et al., 2024).
• GAE requires only standard gradient and masking operations, applies to CNNs, Transformers, and text, and can be decomposed into faithfulness and class-specific contrastiveness for method diagnostics (Vukadin et al., 2024).
• Data-driven nDCG_φ naturally incorporates real score distributions, auto-calibrates for outlier or inlier spread, and can replace ad-hoc graded gains throughout IR (Moniz et al., 2016).

Combining relevance metrics with fluency, diversity, answerability, and other task-specific axes enables modular, composite evaluation. Most methods allow plug-and-play integration: GAE for explainers (Vukadin et al., 2024), nn-IoU for CBIR ranking (Wei et al., 16 Jun 2025), and EXAM for next-generation IR systems (Farzi et al., 2024).

5. Limitations, Controversies, and Open Challenges

Current limitations include:

• Dependence on Embedding Quality and Model Priors: Contextual relevance often crucially depends on the quality and training domain of the feature extractor (e.g. SCAN, BERT, CLIP), inducing potential domain shift or interpretability breakdown (Jiang et al., 2019, Sun et al., 2024).
• Data Sparsity and Annotation Requirements: Some methods (CGDIALOG⁺ for causal discovery in dialogue (Feng et al., 2024)) still require high-quality annotated data; others risk label distribution mismatch.
• Parameter Sensitivity: RS parameters $(\alpha, \beta)$ (Gopalakrishna et al., 2013), data-driven control-point choices (Moniz et al., 2016), or lambda regularization in redundancy-aware summarization (Chen et al., 2021) impact metric behavior.
• Scalability in Knowledge Graph Matching: For nn-IoU, real-time computation requires precomputed neighbor dictionaries; expansion beyond “is-a” graphs or into richer domains remains challenging (Wei et al., 16 Jun 2025).

Controversies persist in how best to balance between token-level, semantic, causal, or answerability-based relevance. For example, reference-based metrics fail in scenarios involving multi-evidence or deep reasoning, leading to systematic under- or over-estimation of true utility (Wang et al., 2022).

6. Extensions and Paradigmatic Shifts

The relevance-based metric landscape is dynamic:

• Reference-free and Training-free Paradigms: Focus is shifting to fully automatic, reference-independent metrics capable of adapting to new domains, e.g., centrality-weighted pseudo-reference approaches in summarization (Chen et al., 2021).
• Knowledge-aware and Gap-focused Evaluation: Explicit modeling of user learning objectives, session-level gap closure, and concept coverage introduce new axes for system optimization beyond topicality (Ghafourian, 2022).
• Multi-modal and Human-centered Relevance: Fusion of language and vision-based metrics and curriculum-style answerability grading represent cutting-edge directions in measuring relevance for attention, captioning, IR, and cognitive modeling (Sun et al., 2024, Farzi et al., 2024).
• Self-supervised Debiasing: Methods such as classifier-based CI testing and self-training structurally expand labeled datasets while maintaining noise reduction, corresponding to scalable human alignment (Feng et al., 2024).

Practitioners are encouraged to select or design relevance metrics aligned with both the expected domain variance and cognitive or practical utility, and to routinely inspect the underlying distributions or disagreement curves, as PRM and data-driven approaches recommend (Demeester et al., 2015, Moniz et al., 2016).

---

Representative Table: Families of Relevance-Based Metrics

Metric Class	Core Mechanism	Sample Papers
Classifier-based	Deep features, LR, binary decision	IDK (Berlot-Attwell et al., 2022), CausalScore (Feng et al., 2024)
Information-theoretic	MI, CI tests, causal inference	CausalScore (Feng et al., 2024)
Disagreement-model	Double-assessor, predicted user relevance	PRM (Demeester et al., 2015)
Data-driven gain	Hermite/pchip, continuous mapping	DCG_φ (Moniz et al., 2016)
Embedding overlap	Cosine in multimodal space	REO (Jiang et al., 2019), MARS (Liu et al., 2021)
Prompt-based models	MLM cloze, pairwise relevance	MetricPrompt (Dong et al., 2023)
Knowledge-aware	Gap closure, concept coverage	RS (Gopalakrishna et al., 2013), KG (Ghafourian, 2022)
Curriculum/QA	Exam question coverage/answerability	EXAM (Farzi et al., 2024)

Each approach reflects a distinct optimization and application locus, and is validated in specific experimental settings and modalities. The evolution of relevance-based metrics is marked by increasing task-specific adaptation, empirical rigor, and alignment with the true functional and cognitive value of information.