Sample-wise Entropy Valley Hypothesis

• Sample-wise Entropy Valley Hypothesis is a framework that quantifies sensor uncertainty with Shannon entropy, forming an entropy valley guiding evidence fusion.
• It employs Dempster–Shafer theory to combine diverse evidences from textual, profile, and citation sensors, ensuring robust and calibrated decision-making.
• Empirical evaluations on bibliometric datasets show significant improvements in precision and MAP over traditional aggregation methods.

The Sample-wise Entropy Valley Hypothesis concerns the characterization and management of uncertainty in multi-source evidence aggregation, with direct implications for the robustness of expert finding systems and general data-fusion frameworks. The concept achieves operational form within expert retrieval by quantifying each sensor’s informativeness and reliability through Shannon entropy on a per-sample basis. This quantification directly modulates the weight and belief assignments when resolving conflicts between heterogeneous evidence streams, using Dempster–Shafer theory as the formal engine for probabilistic reasoning under uncertainty. The interplay between low-entropy (focused) and high-entropy (uncertain) sensor output forms a “valley” in sample-wise entropy, identifying where credible fusion decisions may lie.

1. Multisensor Evidence Extraction

Expert finding in large-scale bibliometric datasets is approached through three distinct sensors, each providing evidence through different modalities:

• Textual-Content Sensor: Utilizes classical IR features (BM25, TF-IDF, Jaccard) between the query and all candidate documents authored by the individual under analysis. Each metric, or “event,” produces a candidate ranking, which are then fused via standard aggregation methods (CombSUM, Borda, Condorcet) to obtain a single normalized fusion score per author.
• Profile-Information Sensor: Derives formal attributes from academic profiles, such as total number of publications, years active, publications per year, and count of publishing venues. Events again correspond to measurements that induce rankings across the candidate set.
• Citation-Graph Sensor: Exploits citation network characteristics—total citations, h-index variants, number of co-authors, PageRank within the publication graph, among others. These yield evidentiary rankings analogous to the previous sensors.

The overall candidate set $C = \{c_1, \ldots, c_{|C|}\}$ frames the “frame of discernment” in Dempster–Shafer theory, with all uncertainty and evidence being referenced against this set (Moreira et al., 2013).

2. Shannon Entropy as Sensor Reliability and Ignorance

To operationalize sample-wise uncertainty, each sensor’s combination of evidentiary events is mapped to a joint distribution over candidate–event pairs. Here, $$p(e,a) = \frac{\mathrm{relevantEvent}(e,a)}{|A| \times |E|}$$, where a “relevant event” is one in which author $a$ receives a nonzero score under event $e$. The Shannon entropy for sensor $S$ is defined as:

$$H(S) = -\sum_{a \in A}\sum_{e \in E} p(e,a)\log_2 p(e,a)$$

with the maximal possible entropy given by $\max H(S) = \log_2(|A| \times |E|)$.

High entropy ($H(S)$ near $\max H(S)$) reflects dispersed, undifferentiated signals—i.e., sensor uncertainty—whereas low values signal focused, high-confidence evidence. This entropy valley over samples governs how much mass is assigned to sensor “ignorance” in subsequent belief fusion (Moreira et al., 2013).

3. Dempster–Shafer Theory for Evidence Fusion

Each sensor’s outputs are encoded as mass functions $m_S$ over the power set $2^\theta$, where only singletons $\{a\}$ (i.e., specific author hypotheses) and the full set $\theta$ (ignorance) have nonzero mass. Specifically:

$$m_S(X) = \begin{cases} \mathrm{Fusion}_S(a), & X = \{a\} \ H(S)/\max H(S), & X = \theta \ 0, & \text{otherwise} \end{cases}$$

with normalization so that $\sum_{a \in A}m_S(\{a\}) + m_S(\theta) = 1$. Two sensors $S_1, S_2$ are fused via Dempster’s rule for evidence combination, where the conflict mass $K$ quantifies total disagreement, and only non-empty intersections yield valid combined mass.

The fusion proceeds in hierarchical stages (e.g., textual ⊕ profile, then ⊕ citation), and the authors are ranked finally by their combined singleton mass, $m_{\mathrm{final}}(\{a\})$ (Moreira et al., 2013).

4. Impact of Entropy Valleys in Fusion Robustness

The critical functional role of the sample-wise entropy valley arises in mass assignment for ignorance. As $H(S)$ increases, $m_S(\theta)$ increases, automatically down-weighting unreliable (high-entropy) sensors and reducing their influence on conflicting singleton mass assignments. This dynamic ensures that no candidate is unjustly suppressed by low-confidence sensors—zero-score authors in one sensor remain eligible if supported by others, due to redistributed “ignorance” mass.

Moreover, the convexity of entropy across samples produces stabilization effects in combination: higher entropy increases the level of fusion-robustness by avoiding high conflict (large $K$), which otherwise would cause mass to be distributed thinly or erratically among candidates. This self-calibrating aspect minimizes factors such as hypersensitivity to any individual sensor or event (Moreira et al., 2013).

5. Empirical Evaluation and Comparative Performance

Experiments on the Proximity and Enriched DBLP datasets, using queries and gold labels from ArnetMiner, assess P@5, P@10, P@15, P@20, and Mean Average Precision (MAP) under varying sensor configurations and fusion schemes.

Fusion Method Performance (Proximity Dataset)

Method	P@5	P@10	P@15	P@20	MAP
D-S + CombSUM	0.7538	0.7000	0.6256	0.5769	0.4402
D-S + Condorcet	0.7538	0.6385	0.5846	0.5615	0.4905
CombSUM alone	0.3385	0.3308	0.3385	0.3115	0.3027
Condorcet alone	0.4615	0.4538	0.3846	0.3538	0.2874

Relative improvements in MAP reach 62–71% over standard aggregation baselines. Similar patterns are observed for best-performing sensor pairs (notably text + citation) and for the enriched dataset, validating the criticality of entropy-driven fusion (Moreira et al., 2013).

6. Advantages Over Supervised Approaches and Broader Implications

The entropy-modulated Dempster–Shafer fusion mechanism—without relying on manually-tuned sensor weights or labeled training data—achieves MAP that matches or exceeds supervised Learning-to-Rank methods (SVM map, SVM rank) when applied to identical feature representations. The sample-wise entropy valley ensures scalable, label-free, and robust evidence fusion for large bibliographic graphs.

A plausible implication is that the entropy valley mechanistically identifies the “trust zone” of each sensor for each sample, permitting fusion frameworks to dynamically allocate belief and ignorance in a conflict-aware, sample-adaptive manner.

7. Related Methodologies and Generalization

The foundational framework emerged in the context of academic expert finding, but is extensible to any application where multiple, heterogeneous evidence streams must be fused resolutely in the presence of uncertainty and variable reliability. The role of sample-wise entropy extends naturally to settings involving ensemble learning, multi-view data mining, and distributed sensor networks, wherever principled mass assignment for conflicting evidence is required.

By formalizing entropy valleys, the approach avoids both overconfident fusion (overfitting to any one unreliable sensor) and indecision (failure to capitalize on confident, low-entropy sources). Thus, it represents a rigorously justified blueprint for fusing uncertain, high-dimensional evidence (Moreira et al., 2013).