Cross-Image Leakage in Vision Systems

• Cross-image information leakage is defined as the unintended influence of one image's features on another, as seen with increased merged caption rates in LVLMs.
• It is characterized by formal metrics such as modality, coverage, and degree (hard/soft), quantified using cosine similarity and leakage scores in benchmarks.
• Mitigation strategies include inference-time masking, dataset sanitation, and embedding obfuscation to preserve privacy and ensure reliable evaluations.

Cross-image information leakage refers to the phenomenon where information from one image influences or contaminates the processing, representation, or decision-making associated with another image, typically violating the expectation of strict independence across samples. This leakage can occur in various contexts—including deep vision-LLMs (LVLMs), large-scale visual datasets, image embeddings, camera fingerprinting, and face recognition descriptors—and manifests as semantic, statistical, or task-driven entanglement. Such leakage degrades benchmark reliability, privacy guarantees, interpretability, and the intended invariance properties of modern vision systems.

1. Formal Definitions and Taxonomy

Cross-image information leakage admits rigorous, task-specific characterizations:

• Interference in Multi-image Models: In LVLMs handling sets of images $\{I_1, I_2\}$, leakage arises when outputs for one image are erroneously influenced by visual cues from others—operationalized by increased probability of selecting "merged" captions (describing both images) over "target-specific" ones in a forced-choice setting. The leakage metric is $C = R_m - R_s$, where $R_m$ and $R_s$ are merged-caption selection rates in multi- and single-image settings, respectively. If $C > 0$, cross-image leakage is present (Park et al., 19 Aug 2025).
• Visual Dataset Leakage: Leakage is formally categorized along:
  • Modality: Image-only (test images appear in training, no labels) or full (both images and labels/annotations in both train/test).
  • Coverage: Intra-dataset (within dataset splits) or inter-dataset (test images in train pool of another dataset).
  • Degree: Hard (pixel-identical duplication) or soft (near-duplicate via crops, color-jitter, etc.) (Ramos et al., 24 Aug 2025).
• Semantic Embedding Leakage: For an image-encoder $f_V : \mathcal{X} \rightarrow \mathbb{R}^m$ and attacker-encoder $f_A : \mathcal{X} \rightarrow \mathbb{R}^n$, semantic neighborhood preservation means an attacker, via linear alignment, can retrieve semantic tags or captions from victim embeddings $e_V$ without inversion to the original image (Chen et al., 30 Jan 2026).
• PRNU-Based Forensic Leakage: For $L$ images $\{Y^{(i)}\}$, the estimated fingerprint $\hat{K}$—derived from all images—can leak global statistics (mutual information) and enable membership inference (determining if a candidate image contributed to $\hat{K}$) (Fernández-Menduiña et al., 2020).
• Face-Embedding Leakage: Supposedly "identity-only" face descriptors $e = E(x)$ retain auxiliary information. Non-ID attributes, pose, histograms, and even image reconstructions become extractable, revealing comprehensive cross-image information (Li et al., 2023).

2. Empirical Manifestations and Impact

Cross-image leakage has been empirically demonstrated in numerous contexts:

• LVLMs and Multi-image Tasks: Models exhibit significant performance loss when tested on multi-image tasks: outputs blend semantic content, with the cross-image leakage score $C$ often substantially $>0$ (Park et al., 19 Aug 2025).
• Visual Dataset Benchmarks: Systematic analysis reveals intra- and inter-dataset hard leakage rates up to $1.9\%$ (e.g., ImageNet-test, COCO-val2017) and soft leakage rates up to $3\%$ intra-dataset and $7.9\%$ inter-dataset (e.g., via LAION-400M). Such overlap inflates evaluation scores and misrepresents model generalization (Ramos et al., 24 Aug 2025).
• Semantic Embedding Attacks: Attackers recover nontrivial semantic content from compressed image embeddings with F1 scores $\gtrsim 0.5$ for large neighborhoods ($m=50$), and substantial ROUGE-L scores (40–60) for tag-based text reconstructions, even when originating from opaque or black-box embedding APIs (Chen et al., 30 Jan 2026).
• Camera PRNU Fingerprints: For $L=26$ images, the mutual information lower bound (ILB) ranges from $0.80$ to $1.92$ bits per pixel (bpp), with membership inference ROC AUC often $>0.95$. Even with $L=50-100$ images, extracted fingerprints retain substantial recoverable image content (Fernández-Menduiña et al., 2020).
• ID Face Embedding Inversion: Non-ID features (e.g., smile, hat, eyeglasses) are predictable at $>90\%$ accuracy; pose and lighting are recovered nearly as reliably as from the image itself. Photorealistic inversion using StyleGAN2 achieves identity similarity (cosine on held-out embedding) up to $0.77$ (Li et al., 2023).

3. Methodologies for Detection and Quantification

Quantifying and detecting cross-image information leakage requires task-specific but rigorous techniques:

• Retrieval-Based Detection in Datasets: To identify leakage, each test image $x_q$ is mapped to an embedding via a fixed encoder (e.g., CLIP). Comparing to a candidate training image $x_o$, cosine similarity $s_{q,c}$ is thresholded to distinguish hard ($\geq 0.98$) and soft ($0.95 \leq \cdot < 0.98$) matches. Automated large-scale screening identifies and rates leakage in evaluation sets (Ramos et al., 24 Aug 2025).
• Forced-Choice Captioning Metrics: In LVLMs, constructing "target-specific," "distractor," and "merged" captions allows direct comparison in single-image ($D_s$) and multi-image ($D_m$) settings, yielding leakage scores $C$ (Park et al., 19 Aug 2025).
• Membership Inference and Mutual Information: For PRNU fingerprints, two metrics are used: (i) the mutual information between the set of images and the estimated fingerprint, giving a global lower-bound on leakage; (ii) statistical tests (e.g., Neyman–Pearson, normalized cross-correlation) for identifying individual membership in estimation sets (Fernández-Menduiña et al., 2020).
• Semantic Alignment and Neighborhood F1: In semantic embedding leakage, linear alignment and neighborhood recall/precision metrics assess preservation of semantics post-embedding transformation, connecting leakage directly to recoverable tag/caption content (Chen et al., 30 Jan 2026).
• Attribute Prediction and Generative Inversion: In face embeddings, regression from embeddings to attribute space and optimization-based inversion via GAN priors quantitatively and qualitatively reveal the extent of non-ID information retention (Li et al., 2023).

4. Mitigation Strategies

A breadth of countermeasures target different sources and effects of cross-image information leakage:

• Architecture-Agnostic Inference (FOCUS): The FOCUS strategy masks non-target images with random noise at inference, executes per-image forward passes, performs contrastive subtraction using a noise-only context, and aggregates logits before token sampling. This method suppresses cross-image entanglement without retraining or architecture changes; it delivers up to 32 point increases in Image Score and 30 point increases in Group Score on challenging multi-image benchmarks (Park et al., 19 Aug 2025).
• Dataset Sanitation: Automatic duplicate and near-duplicate detectors (embedding-based retrieval with strict thresholds) are essential prior to model development and publication. Removing or masking matches from training or test pools prevents test-train overlap (Ramos et al., 24 Aug 2025).
• Embedding Obfuscation: Semantic obfuscation (injecting noise/randomization) and differential privacy techniques, such as randomized response or adversarial training (to minimize mutual information between embedding and sensitive attributes), disrupt neighborhood structure exploited in semantic leakage (Chen et al., 30 Jan 2026).
• Forensic Fingerprint "Deleaking": For PRNU, maximizing the number of flat-field images in estimation, applying compression or whitening to fingerprints, and operating in the encrypted domain reduce information content and membership inference viability (Fernández-Menduiña et al., 2020).
• Embedding Regularization and Compression in Face Rec: Adversarial training with auxiliary attribute predictors, structured-loss penalties to enforce invariance to nuisance variables, and embedding quantization or noise injection can mitigate leakage of non-ID attributes (Li et al., 2023).

5. Broader Implications and Privacy Considerations

Cross-image information leakage has ramifications extending from scientific evaluation to real-world deployment and privacy:

• Benchmark Validity: Overlap between evaluation benchmarks and training data—often exacerbated by large-scale web-scraped datasets—systematically inflates reported metrics, undermining progress claims and masking overfitting. Current practice of single-encoder or shallow duplicate checks is inadequate, demanding automated, large-scale retrieval-based validation (Ramos et al., 24 Aug 2025).
• Privacy and Security Risks: Retained semantic or image information in feature vectors or fingerprints can be exploited for membership inference, attribute extraction, or partial scene reconstruction. Embedding-based API and vector search systems (e.g., RAG, recommendation, retrieval systems) are inherently vulnerable unless neighborhood structure is actively disrupted (Chen et al., 30 Jan 2026, Li et al., 2023).
• Capacity and Invariance Trade-offs: In face and object recognition, leakage of nuisance attributes wastes descriptor capacity and degrades downstream performance unless penalized. Claims of exclusive ID encoding are contradicted by both regression and inversion results (Li et al., 2023).
• Forensic Sensitivity: Sharing raw camera fingerprints carries risk of exposing underlying image content, jeopardizing privacy in legal, governmental, and journalistic uses. The imperative is to "never reveal $\hat{K}$ in the clear" and to require privacy-preserving estimation protocols (Fernández-Menduiña et al., 2020).

6. Recommended Best Practices and Future Directions

Mitigating cross-image information leakage requires comprehensive, systemic changes spanning the ML pipeline:

• Pre-publication Analysis: Always conduct automated retrieval-based leakage assessments on any proposed benchmark or training set, with thresholds calibrated for both hard and soft near-duplicate matches. Remove all duplicates from training, and either mask or exclude from test if removal from train is infeasible (Ramos et al., 24 Aug 2025).
• Inference-time Decoding and Adversarial Objectives: Leverage inference-time logit aggregation, noise-masking (as in FOCUS), and adversarial unlearning frameworks to minimize cross-image and cross-domain information entanglement (Park et al., 19 Aug 2025, Li et al., 2023).
• Embedding Privacy Protocols: Implement semantic obfuscation, privacy-aware alignment, and watermarking/keyed-encoder schemes before sharing or deploying embedding-based services (Chen et al., 30 Jan 2026).
• Forensic and Face Rec Defenses: Never expose raw statistical fingerprints; use large-scale, flat-fielded estimation for PRNU and actively penalize or compress non-ID attribute retention in face-recognition deployment (Fernández-Menduiña et al., 2020, Li et al., 2023).
• Ongoing Community Audits: Foster open-source tools and protocols for rigorous, reproducible leakage screening; support versioned, sanitized dataset releases with clear provenance.

The persistence of cross-image information leakage, even under compressed, approximate, or obfuscated representations, signals a foundational challenge: effective utility and invariance are in structural tension with privacy and independence. Advancing the field will require embedding-level privacy guarantees, robust evaluation protocols, and a reassessment of training and benchmarking paradigms across vision-centric machine learning.