Training for Token-Utility Maximization in Late Interaction Retrieval

Determine whether explicitly training ColBERT-style MaxSim late-interaction retrieval models with objectives that maximize the utility of each document token in multi-vector representations leads to improved retrieval performance across multimodal datasets.

Background

The paper analyzes how often document tokens participate in MaxSim matches during late interaction and observes that, in multimodal settings, only about 1% of tokens are typically active in a single evaluation pass. This underutilization suggests significant redundancy and motivates developing methods that encourage broader token participation.

Empirical analysis on MSR-VTT shows strong correlations between retrieval performance and evenness measures (e.g., Coefficient of Variation and Gini) of the distribution of maximum similarity matches across document positions. Based on these correlations, the authors hypothesize that training objectives which explicitly increase token-level utilization could improve retrieval effectiveness, but they do not test such objectives here due to scope limitations.