Vision In-Context Learning (VICL)

• Vision In-Context Learning (VICL) is a paradigm that uses demonstration image–output pairs to condition frozen vision models, enabling task adaptation without retraining.
• It employs prompt-driven adaptation through advanced selection, fusion, and arrangement techniques to perform diverse visual tasks like segmentation, detection, and image editing.
• Recent advances include principled prompt ranking, multi-prompt collaborative architectures, and applications to video, medical imaging, and robotics, driving innovations in adaptive visual intelligence.

Vision In-Context Learning (VICL) is a paradigm in which large pre-trained vision models generalize to new tasks by conditioning on a small number of visual demonstrations—image–output pairs—provided at inference time, without gradient updates or retraining. Inspired by the success of in-context learning in LLMs, VICL enables both discriminative and generative visual tasks (e.g., segmentation, detection, image editing, and colorization) to be performed via prompt-based input and analogy-driven reasoning. Modern VICL approaches leverage visual backbones such as masked autoencoders (MAE-VQGAN), diffusion transformers (DiT), and multimodal vision-LLMs, using prompt selection, fusion, and arrangement techniques to maximize context utility. Over the past three years, VICL has yielded new architectures, prompt selection frameworks, multi-prompt fusion strategies, and extensions to video, personalization, and medical imaging, establishing itself as a central research avenue for adaptive, generalist visual intelligence.

1. Formal Framework and Core Principles

The VICL process centers on a frozen visual foundation model (VFM)—often a masked inpainting or diffusion model—conditioned on a prompt comprising $K$ demonstration pairs $\{(x_i, y_i)\}_{i=1}^K$ and a query image $x_q$. The model is tasked to output $\hat y_q = f_\theta(\{(x_i, y_i)\}, x_q)$, where $f_\theta$ denotes the unmodified pre-trained network. Standard VICL input is arranged as a $2\times2$ grid: $$X = \begin{bmatrix} x_{c_1} & y_{c_1} \ x_q & \text{[MASK]} \end{bmatrix}$$ The model’s objective is typically to inpaint or reconstruct the [MASK] region matching the label or output for $x_q$. No task-specific re-training is performed; generalization comes purely from context and model capacity (Sun et al., 2023, Xie et al., 27 Mar 2025).

Distinctive features of VICL include:

• Prompt-driven adaptation: Sampling and arrangement of visual demonstrations are the principal means for task conditioning.
• Unified model interface: The same VFM supports diverse tasks (segmentation, detection, image-to-image translation) by varying the prompt content without altering model weights.
• Zero/few-shot behavior: Even with a single demonstration (one-shot VICL), models show considerable generalization, and benefits accrue with more or more diverse prompts (Wang et al., 30 Apr 2025, Liao et al., 15 Jan 2026).

2. Prompt Selection: From Similarity to Global Ranking

Prompt selection is the dominant factor in VICL performance. Early methods used CLIP-based or pixel-level retrieval to choose the demonstration most visually similar to the query, but later research recognized the need for more principled strategies.

• Pixel-Level Retrieval: Pixelwise cosine similarity between query and candidates yields finer semantic alignment than global features. For instance,

$s_k(i, j) = \hat f_q(i) \cdot \hat f_s(k, j)$

where $f_q$ and $f_s$ are spatial features (Sun et al., 2023).

• Ranking and List-Wise Aggregation: Transformer-based listwise rankers optimize a composite loss (margin, NDCG, regression) over subsets of candidate prompts and aggregate pairwise preferences to produce a consistent global ranking. The Partial2Global framework employs such a ranker with least-squares consistency aggregation to ensure globally optimal prompt selection (Xu et al., 2024).
• Reliable and Holistic Selection: RH-Partial2Global introduces a jackknife conformal prediction filter to identify candidates with strong alignment between visual similarity and in-context utility, then ensures uniform pairwise comparison coverage with combinatorial covering designs. This two-pronged approach improves robustness and generalization, as confirmed by mIoU and MSE gains over Partial2Global (Wu et al., 30 Sep 2025).
• Task-Level Prompt Sharing: Contrary to the intuition that per-sample optimal prompts must be sought, it is empirically observed that a single prompt set often suffices for most samples of a task. Task-level prompt search (Top-K, Greedy) dramatically reduces computational costs while achieving near-optimal accuracy (Zhu et al., 15 Jan 2025).

3. Prompt Fusion and Multi-Prompt Collaborative Architectures

While single-best prompt selection may discard useful complementary cues, recent frameworks focus on integrating multiple high-quality examples:

• Prompt Condensation: The Condenser plugin aggregates $K$ prompts by patch-wise cross-attention, mapping each query location to a weighted sum of aligned prompt patches. Combined with a pre-alignment loss toward the query’s own features and token-level supervision, Condenser delivers both accuracy and computational efficiency compared to late-stage ensembling (Wang et al., 30 Apr 2025).
• Multi-Faceted Fusion: Collaborative approaches (e.g., MULTI-VQGAN) form multiple prompt groupings (main, high-similarity, low-similarity), process them through parallel transformer branches, and fuse their features at select depths by learned cross-attention modules. This hierarchical strategy offers richer, disentangled guidance without collapsing all signals into one as in mean-pooling (Liao et al., 15 Jan 2026).
• Arrangement Modules and Bidirectional Fine-Tuning: Arrangement-specific adapters (MLPs) capture geometric priors of different $2×2$ layouts. Joint fine-tuning with bidirectional swap steps (reversing query/prompt roles) enhances collaboration between fusion, adapters, and the inpainting model, yielding superior and more robust mask and colorization outputs (Liao et al., 15 Jan 2026).
• Training-Free Multi-Prompt Smoothing: The PANICL approach performs k-nearest neighbor smoothing of codebook assignment scores across many prompts, reducing single-prompt overfitting and improving robustness to domain and label-space shifts without any additional model training (Zhang et al., 26 Sep 2025).

4. Application Domains and Advanced Adaptation

VICL underpins advances across recognition, generation, editing, and control:

• Personalized Vision and Edit Transfer: In the PICO framework, a four-panel input (exemplar and query-image pairs) guides a diffusion transformer (DiT) to transfer visual relations (e.g., personalized edits or non-rigid transformations) to arbitrary new inputs, with attention-guided seed scoring stabilizing outputs (Jiang et al., 29 Sep 2025, Chen et al., 17 Mar 2025).
• Medical Imaging: In retinal OCT, the Retinalizer model uses VICL with a U-Net backbone and pairwise-convolution context fusion to execute 23 tasks (semantic segmentation, denoising, super-resolution, inpainting) from a single pool of context examples, with random recoloring augmentation driving cross-task and cross-domain adaptation (Negrini et al., 18 Jun 2025).
• Robotic Manipulation with Vision-Language Context: OmniVIC combines retrieval-augmented memory with VICL-style prompts composed of vision, language, proprioceptive, and force-torque cues. A vision-LLM generates context-aware impedance gains for safe, adaptive manipulation, outperforming fixed policies in both simulation and real-world contact-rich tasks (Zhang et al., 20 Oct 2025).
• Video and Cross-Task VICL: Video VICL extends the paradigm to self-supervised Transformer models conditioned on demonstration clips for zero-shot video imitation, with clear scaling laws for parameter and dataset size (Zhang et al., 2024). T2T-VICL investigates VICL where support and query examples come from distinct low-level vision tasks, using entire pipelines for implicit prompt generation, selection, and perceptual score-based ranking (Xia et al., 20 Nov 2025).

5. Vision-Language, Long-Context, and Dataset Considerations

• Vision–Language In-Context Learning: Visual ICL is extended to Large Vision-LLMs (LVLMs) by converting visual context into concise, intent-oriented textual summaries, resolved via intent-driven retrieval and composition frameworks. Efficient VICL solutions for LVLMs alleviate token limits and cross-modal mismatch, and enable in-context unlearning via counter-labeled prompts (Zhou et al., 2024, Wang et al., 12 May 2025).
• Efficient Long-Context MLLMs: VisInContext renders long text context into compact images, extracts visual tokens, and feeds them to multimodal LLMs, achieving 6–40× FLOP reductions and enabling significant scaling of in-context lengths for efficient few-shot and document QA scenarios (Wang et al., 2024).
• Dataset Construction and Diversity: Successful generalization for personalized and cross-task VICL hinges not just on scale but on task diversity and careful arrangement of demonstration examples. Frameworks such as VisRel curate compact yet highly diverse prompt pools spanning recognition, geometry, generative editing, and multi-style manipulation, ensuring that the VICL model’s representation space supports emergent and open-ended tasks (Jiang et al., 29 Sep 2025, Xia et al., 20 Nov 2025).

6. Evaluation Protocols and Empirical Insights

• VICL is benchmarked on segmentation (mean IoU), detection (mIoU or AP), colorization (MSE), generative metrics (PSNR, SSIM, LPIPS), cross-domain robustness, and domain generalization (e.g., medical OCT datasets and held-out vendors) (Xie et al., 27 Mar 2025, Liao et al., 15 Jan 2026, Negrini et al., 18 Jun 2025).
• Ablation studies across fusion strategies, arrangement modules, covering design sampling, and conformal filtering demonstrate that fusing prompts—both at the architecture and algorithmic levels—substantially outperforms single-prompt or naive ensemble baselines.
• Key practical findings include the trivial cost of scaling prompt context in certain architectures, the improved computational efficiency of prompt condensation over voting ensembles, and the importance of prompt reliability and diversity over mere similarity.

7. Limitations, Open Problems, and Future Directions

• Prompt Reliability: Visual similarity does not guarantee in-context utility; advanced filtering (RH-Partial2Global, conformal prediction) is needed to weed out “false friends” in retrieval (Wu et al., 30 Sep 2025).
• Scaling and Complexity: Larger context pools and multi-branch architectures introduce computational and selection overheads, motivating research into efficient fusion, adaptive sampling, and learned covering designs (Liao et al., 15 Jan 2026).
• Task Generalization: Achieving true cross-task, cross-modal, and “open vocabulary” VICL requires both richer prompt pools and architectures robust to out-of-domain compositionality (Xia et al., 20 Nov 2025, Jiang et al., 29 Sep 2025).
• Unlearning and Safety: In-context unlearning via negative demonstration prompts and generalized safety constraints in VICL-driven robotics offer new tools for privacy and reliability but remain underexplored (Zhou et al., 2024, Zhang et al., 20 Oct 2025).
• Transfer to Video/Sequential Data: Hierarchical fusion, spatiotemporal tokenization, and retrieval-augmented strategies in video ICL represent a promising direction for extending VICL to embodied and multi-step control scenarios (Zhang et al., 2024).

In summary, Vision In-Context Learning defines a rapidly advancing regime in computer vision, offering remarkable flexibility for task-agnostic adaptation and task transfer. The field continues to deepen through more principled prompt selection mechanisms, collaborative multi-prompt architectures, and extensions into diverse domains and modalities (Sun et al., 2023, Xu et al., 2024, Wang et al., 30 Apr 2025, Liao et al., 15 Jan 2026, Wu et al., 30 Sep 2025, Zhang et al., 20 Oct 2025).