In-Context Learning Prompt Retrieval

• In-context learning prompt retrieval is the process of selecting and ordering demonstration examples from a labeled pool to configure a frozen model for downstream tasks.
• It employs diverse retrieval algorithms such as unsupervised similarity, supervised ranking, combinatorial search, and reinforcement learning, achieving performance gains like a 6–8% improvement in segmentation mIoU.
• Optimizing prompt composition and sequencing, including attention to diversity and context constraints, significantly narrows the gap to fully supervised or fine-tuned baselines.

In-context learning (ICL) prompt retrieval is the process of selecting and assembling demonstration examples for inclusion in a prompt so that a frozen pre-trained model (language or vision) can be configured to solve a downstream task without parameter updates. The retrieval mechanism directly determines ICL performance, with selection criteria ranging from surface similarity to downstream task utility, and retrieval operations spanning unsupervised heuristics, supervised ranking, combinatorial search, and neural policy agents. Across NLP and vision, recent work demonstrates that meticulous prompt retrieval—with explicit attention to relevance, diversity, and even ordering—substantially narrows or even closes the gap to fully supervised or fine-tuned baselines.

1. Formal Problem Formulation

Prompt retrieval in ICL is typically formalized as selecting a subset $P = \{(x_j, y_j)\}_{j=1}^K$ from a labeled pool $D$ of demonstration pairs. Given a test instance $x_q$, the aim is to maximize the task accuracy $\mathcal{S}$ or minimize loss $\mathcal{L}$ of a frozen model $f$ under the prompt:

$$P^* = \underset{P \subseteq D, |P|=K}{\arg\max} \, \mathcal{S}(f(P, x_q), y_q)$$

This is subject to a context window constraint. In classification or structured prediction, $f$ is a language or vision model conditioned on the prompt examples. In dense prediction or detection, $f$ may further partition $P$ into image patches or memory cells (Zhu et al., 15 Jan 2025, Zhang et al., 2023, Sun et al., 2023, Balažević et al., 2023, Suo et al., 2024).

In the task-level regime, the optimal $P$ is found by minimizing the expected loss over a held-out validation set,

$$P^* = \arg\min_P \sum_{(x_i, y_i) \in D \setminus P} \mathcal{L}(f(P, x_i), y_i)$$

offering significant computational savings when prompts are to be shared across many queries (Zhu et al., 15 Jan 2025).

2. Retrieval Algorithms and Scoring Mechanisms

2.1 Unsupervised Similarity-based Retrieval

Unsupervised schemes use pretrained feature encoders (e.g., CLIP, ViT, SBERT) to compute cosine similarity scores:

$s(x_q, x_j) = \langle \phi(x_q), \phi(x_j) \rangle$

The top-$K$ nearest neighbors form $P$. This is widely adopted in vision ICL (e.g., segmentation, detection, colorization), typically yielding a 6–8% absolute gain in mIoU over random selection, and is robust to backbone choice (Zhang et al., 2023, Suo et al., 2024). In retrieval-augmented LLMs, BM25 or embedding-based kNN serves as a default (Parry et al., 2024, Rubin et al., 2021, Ng et al., 18 Nov 2025).

2.2 Supervised, Utility-informed Retrieval

Supervised retrievers learn a parametric function $s_\theta(x, z)$ reflecting task-specific relevance—specifically, whether inclusion of $z$ in the prompt for $x$ leads to correct prediction:

$$r(x, z) = \mathbb{1} \{\text{LLM\_pred}(x{:}z) = y(x)\}$$

A bi-encoder or cross-encoder is trained using positives ($r=1$) and hard negatives ($r=0$), using contrastive loss (InfoNCE) or binary cross-entropy. At inference, fast maximum inner product search (bi-encoder, for scalability) or reranking (cross-encoder, for precision) retrieves prompt sets (Parry et al., 2024, Rubin et al., 2021).

Supervised approaches systematically outperform unsupervised ones: e.g., on MTop, dual-encoder EPR achieves 64.2% vs. 52.9% for BM25, and matches or exceeds BM25-Oracle (Rubin et al., 2021). In vision, SupPR outperforms UnsupPR by $>1$ mIoU point and remains robust under domain shift (Zhang et al., 2023).

2.3 Combinatorial and Task-Level Search

Sample-level optimal prompt search is computationally prohibitive. Recent work exploits the empirical observation that many test samples share the same near-optimal prompt. Greedy additive or top-K scoring schemes identify a single task-level prompt $P^*$, used for all queries, recovering near-oracle accuracy with 10–50$\times$ speedup and negligible loss ($<6\%$ mIoU drop), especially when the validation pool is representative (Zhu et al., 15 Jan 2025).

2.4 Policy and Reinforcement Learning Agents

Reinforcement learning agents, trained via policy gradient, adaptively select the optimal prompt set conditioned on query features, using the model’s downstream score (e.g., IoU) as reward (Suo et al., 2024). These agents efficiently balance relevance and diversity, yielding practical annotation savings and improved segmentation accuracy.

3. Diversity, Order, and Structural Factors in Prompt Construction

Diversity vs. Similarity

Purely similarity-based retrieval may yield prompts that are overly redundant. Incorporating diversity (e.g., via nearest plus farthest cluster exemplars) exposes the model to a wider range of contexts, enhancing generalization and segmentation (e.g., SegGPT + SCS: $59.5$ mIoU vs. $52.7$ for random 5-shot) (Suo et al., 2024).

Order Sensitivity

Contrary to conventional wisdom, the ordering of in-context examples induces as much variance in model performance as the composition of the example set itself (mean selection/ordering sensitivity ratio $r \approx 1.14$). Efficient ordering search using a small dev set ($|D_{\mathrm{dev}}|\geq 250$) and up to 128 permutations can recover $>95\%$ of the oracle performance, underscoring the intertwined importance of both retrieval and sequencing (Li et al., 12 Nov 2025).

Few-shot and Chain-of-Thought Retrieval

In hybrid reasoning or table–text QA, prompt layout and type-aware structuring matter significantly: the HRoT strategy reconstructs minimal sub-tables and explicitly triggers “retrieval thinking” before final reasoning, reducing hallucination and improving EM/F1 over standard CoT or fully-supervised SOTA by $\sim$2 pts (Luo et al., 2023).

4. Domain-specific Retrieval: Vision, Language, Hybrid QA, Dialogue

Vision and Scene Understanding

State-of-the-art visual ICL retrieval integrates pixel-level similarity, patchwise memory banks, and (optionally) cross-image context pretraining for dense tasks (segmentation, depth). Vision prompts often include spatial arrangements and multiple fusion strategies, ensembled via majority voting to activate diverse model knowledge (Sun et al., 2023, Zhang et al., 2023, Balažević et al., 2023).

Approaches such as Hummingbird couple nearest-neighbor support feature retrieval with contextually pre-trained encoders, achieving strong results without any parameter adaptation and fast data-efficient configuration (Balažević et al., 2023).

Language and Financial Retrieval

In document and chunk ranking for financial filings, PRISM retrieves top-$k$ semantically similar (query, doc/chunk) examples using embedding models (OpenAI TE3), scored by L2 distance in a FAISS store. Best accuracy (NDCG@5 = 0.71818) is reached by combining ICL-based example retrieval with carefully engineered task instructions, but using ICL sparsely (only at the document level) to avoid prompt overload (Ng et al., 18 Nov 2025).

Dialogue and Corrupted Demos

In persona-based dialogue, retrieval using random context-similar demos not only outperforms embedding-based selection and exact match, but also provides increasing gains as $k$ grows—even when demos are heavily corrupted (scrambled or with random responses), a finding not explained by standard induction head mechanisms (Pu et al., 2024). This suggests LLMs exploit global token distribution statistics from the demos, rather than local n-gram induction alone.

5. Key Empirical Results and Performance Benchmarking

Method/Setting	NLP Structured QA (EM)	Vision (Seg. mIoU)	Financial NDCG@5	Dialogue Quality (s₍qual₎)
Random retrieval	1.7–8.9 (Break–SMCal)	27.56 (Seg.)	0.6556	0.160–0.487
Unsupervised retrieval	21.6–52.9	33.56–41.02	0.6669	0.156–0.500
Supervised (bi-encoder)	31.9–64.2	35.56	—	—
Task-level greedy	—	36.8	—	—
PRISM best config	—	—	0.7182	—
HRoT (4-shot)	46.2	—	—	—
Random $k$-shot (dialogue)	—	—	—	$\uparrow$ with $k$, robust to corruption

On semantic parsing, entity-to-SQL, and dataflow tasks, bi-encoder EPR outperforms surface or paraphrase-based baselines by 3–8 pts (Rubin et al., 2021). In segmentation, pixel-level visual prompt selection with spatial fusion/ensemble (prompt-SelF) reaches 41.02 mIoU, surpassing meta-learning OSLSM (40.80) (Sun et al., 2023). On hybrid QA, HRoT few-shot achieves EM/F1 of 46.2/46.9, beating prior SOTA (44.2/44.8) (Luo et al., 2023). PRISM outperforms comparable training-free baselines for document ranking (NDCG@5 0.71818) by integrating embedding-based example retrieval in its system prompt (Ng et al., 18 Nov 2025).

6. Advances, Limitations, and Open Questions

Prompt retrieval for ICL leverages both classical IR methods and neural retrieval architectures. Supervised, task-utility-driven scoring provides substantial improvements over unsupervised similarity, yet currently most systems treat each in-context demonstration independently during retrieval and labeling. Modeling interactions among prompt examples and jointly optimizing for subset composition remains an open challenge (Rubin et al., 2021, Parry et al., 2024).

Recent work overturns the assumption that only the choice of exemplars matters: the order of those demonstrations is equally impactful, motivating joint retrieval–ordering pipelines (Li et al., 12 Nov 2025). In vision and hybrid QA, encoding hierarchical relationships, context-aware reasoning, and hybrid prompt formatting are critical for accurate numerical, tabular, and multi-modal reasoning (Luo et al., 2023, Balažević et al., 2023).

Annotation bottlenecks are addressed by compressed, diverse candidate pools via clustering (reducing storage/labeling by 500$\times$ or more) and by reinforcement-learned retrieval agents. Nonetheless, reliability in the ultra-low-shot regime, scaling up memory indexes, and interpretability of LLMs’ reliance on context vs. parameter memory require further research (Balažević et al., 2023, Kahardipraja et al., 21 May 2025).

• "HRoT: Hybrid prompt strategy and Retrieval of Thought for Table-Text Hybrid Question Answering" (Luo et al., 2023)
• "Exploring Effective Factors for Improving Visual In-Context Learning" (Sun et al., 2023)
• "What Makes Good Examples for Visual In-Context Learning?" (Zhang et al., 2023)
• "Exploring Task-Level Optimal Prompts for Visual In-Context Learning" (Zhu et al., 15 Jan 2025)
• "Towards In-context Scene Understanding" (Balažević et al., 2023)
• "PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval" (Ng et al., 18 Nov 2025)
• "Order Matters: Rethinking Prompt Construction in In-Context Learning" (Li et al., 12 Nov 2025)
• "In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval" (Parry et al., 2024)
• "Learning To Retrieve Prompts for In-Context Learning" (Rubin et al., 2021)
• "Visual Prompt Selection for In-Context Learning Segmentation" (Suo et al., 2024)
• "The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation" (Kahardipraja et al., 21 May 2025)
• "Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of In-Context Learning for Persona-based Dialogue Generation" (Pu et al., 2024)