Self-Supervised OvO Selection

• Self-Supervised Output-vs-Output (OvO) selection is a technique that evaluates and ranks model outputs by comparing them directly without relying on external ground truth.
• It employs pairwise and tournament-style evaluations with internal quality scores to optimize prompts and improve segmentation decisions efficiently.
• This framework is applied in large language model prompt optimization and video segmentation, achieving higher performance and cost efficiency over traditional fusion methods.

Self-supervised Output-vs-Output (OvO) selection refers to a family of evaluation and selection procedures that identify the optimal output from among multiple model predictions by systematically comparing them against each other—entirely without requiring access to external supervision or ground-truth references. This paradigm underpins efficient, scalable model optimization in diverse domains including prompt engineering for LLMs and unsupervised video object segmentation, where either explicit ground truth is impractical to provide or the notion of correctness is inherently ambiguous.

1. Formalization of the Output-vs-Output Selection Principle

The OvO selection approach operationalizes the model evaluation and optimization process by comparing the outputs produced under differing conditions or parameterizations in a pairwise fashion. In the context of self-supervised prompt optimization for LLMs, the objective is to discover a prompt $P^*$ from the prompt space $\mathbb{P}$ that yields maximal expected performance on a task distribution $D$ over input–output pairs $(x, y^*)$. The formal goal is:

$$P^* = \arg\max_{P\in\mathbb{P}}\,\mathbb{E}_{x\sim D}\left[\phi_\mathrm{eval}(\phi_\mathrm{exe}(P, x))\right]$$

Here, $\phi_\mathrm{exe}(P, x)$ denotes the LLM's output under prompt $P$ for input $x$, and $\phi_\mathrm{eval}(y)$ is an internal, model-based quality score. Ground truth $y^*$ is unavailable in the self-supervised regime. OvO selection implements a pairwise scoring function $S(y_i, y_j)$, typically realized via a model-as-evaluator $E$, which determines which of two outputs better satisfies task requirements. This induces a preference ordering over prompt candidates solely from their outputs, bypassing reference to $y^*$ (Xiang et al., 7 Feb 2025).

In unsupervised video object segmentation, a related procedure selects the output between two passes—one conditioned on motion cues (optical flow) and one on appearance cues (RGB image)—using a “confidence map” to decide which prediction is more reliable for each frame. The OvO selection at test time is based on the sum of per-pixel confidence margins (Cho et al., 2023).

2. Mechanisms of OvO Selection Across Domains

LLM Prompt Optimization

The OvO selection in prompt optimization proceeds via a tournament-style algorithm:

• A small batch $Q = \{x_1, ..., x_m\}$ of $m$ inputs is sampled.
• Candidate prompts $P_i$ generate outputs $y_i = \phi_\mathrm{exe}(P_i, x)$ for all $x\in Q$.
• Each prompt pair $(P_i, P_j)$ is evaluated on the corresponding outputs $(y_i, y_j)$ via an LLM-based evaluator $E$, which returns a binary judgment encoded as $S(y_i, y_j)=1$ if $y_i$ is judged superior.
• The winner is determined using majority vote across $m$ samples for each pairwise match. Ties are resolved randomly or by repeated evaluation (Xiang et al., 7 Feb 2025).

Video Object Segmentation

The “motion-as-option” network introduces OvO selection as an adaptive output selector at inference. Two predictions per frame are generated—using flow (motion) and RGB (appearance) as the motion encoder input. Each prediction produces a per-pixel foreground probability $\omega(p)$. A confidence map $\phi(p)$ is computed by thresholding low-margin probabilities, and the global confidence $\alpha$ is the sum over pixels. The output with higher confidence $\alpha$ is selected as final, frame-wise segmentation (Cho et al., 2023).

3. Algorithmic Structure and Optimization Procedures

The self-supervised OvO selection method in prompt optimization alternates between two phases:

• OvO Tournament (Selection): For current best prompt $P^*$ and a candidate $P'$, outputs for sampled inputs are compared via LLM evaluator $E$ on each input. Majority wins decide if $P'$ replaces $P^*$.
• Prompt Refinement (Optimization): An LLM optimizer $\phi_\mathrm{opt}$ takes the current prompt and its outputs, generating a revised prompt aiming for better alignment with task requirements.

The high-level pseudocode is as follows:

P* ← P₀ A* ← [LLM(P*, x) for x in Q] for t in 1…N: P′ ← φ_opt(P*, A*) A′ ← [LLM(P′, x) for x in Q] wins = 0 for each x in Q: if E(A′[x], A*[x]) judges A′ better: wins += 1 if wins > m/2: P* ← P′ A* ← A′

Only $m=3$ examples per iteration and $N=10$ iterations are required, leading to high sample efficiency (≈30 executions and 30 evaluations) (Xiang et al., 7 Feb 2025).

In video segmentation, both motion and appearance pathways are always enabled at test time, with OvO selection reducing to a simple confidence-score maximization—no additional loss or iterative optimization occurs beyond inference (Cho et al., 2023).

4. Theoretical Rationale and Empirical Properties

The core theoretical insight is that quality differentials between outputs induced by different prompts (or input modalities) can be assessed internally—by strong models acting as their own evaluators. This leverages:

• Output as Evaluation Reference: Reasoning traces and answers under divergent conditions manifest differences detectable via a sufficiently capable LLM evaluator.
• Output as Optimization Guidance: By iteratively preferring outputs that better align with requirements, OvO mimics a best-arm identification problem in multi-armed bandits, progressively steering toward optimal prompts.

Convergence toward high-quality candidates is ensured under the assumption that evaluator judgments are positively correlated with the latent ground-truth quality. Random tie-breaking and repeated rounds mitigate the risk of local optima (Xiang et al., 7 Feb 2025).

In the video segmentation context, using OvO selection at inference consistently outperforms input-fusion and feature-fusion approaches, as demonstrated in benchmark ablations (Cho et al., 2023).

5. Benchmark Outcomes and Comparative Analyses

Prompt Optimization

Empirical studies on closed- and open-ended benchmarks establish the superiority of self-supervised OvO selection as implemented in Self-Supervised Prompt Optimization (SPO):

Method	Avg. Perf.	Avg. Cost ($)
Chain-of-Thought	63.8	–
APE	64.8	9.07
OPRO	66.6	4.51
PromptAgent	65.0	2.71
PromptBreeder	64.5	4.82
TextGrad	63.9	13.14
SPO (OvO)	66.9	0.15

SPO matches or exceeds the best baseline (e.g., +0.3 over OPRO) at only 1.1%–5.6% of their compute cost, with enhanced win-rates for smaller models in open-ended settings (Xiang et al., 7 Feb 2025).

Video Segmentation

In adaptive output selection, ablation studies yield the following test-time results (ResNet-101 backbone):

Method	DAVIS G	FBMS J	YTO J
Flow only	86.1	79.9	71.5
Image only	80.0	80.0	73.1
Input/Feature Fusion	83.7–84.6	80.3–80.9	72.8–73.3
Output averaging	84.7	81.0	73.2
Adaptive selection (OvO)	86.1	81.2	73.1

OvO adaptive selection yields the highest or tied-for-best results, with selection frequency for image vs. flow varying by dataset—for DAVIS 2016, flow is selected 96.2% of the time, while on FBMS and YouTube-Objects, images are selected 37.6% and 38.8% of the time, respectively (Cho et al., 2023).

6. Practical Considerations, Hyperparameters, and Limitations

Self-supervised OvO selection is highly efficient:

• Prompt Optimization (SPO):
  • Recommended: $N=10$ iterations, $m=3$ samples/iteration, $k=1$ prompt/candidate per round, 4 repeats per input for tie-breaking.
  • No external supervision required; all scoring is derived from internal comparisons via LLM evaluator prompts.
  • Limitations include dependence on evaluator LLM's biases, absence of formal global optimality guarantees, and reduced effectiveness if the evaluator lacks sufficient capability.
  • Extensions involve larger candidate pools ($k>1$), hybrid methods with occasional output-vs-ground-truth (OvG), rich textual feedback integration, and meta-learning of evaluator prompts (Xiang et al., 7 Feb 2025).
• Video Segmentation:
  • OvO selection hyperparameter $h=0.05$ is robust over $[0.01, 0.1]$.
  • Model is trained only with standard cross-entropy over per-pixel masks; OvO selection acts purely at test time, with no additional self-supervised loss (Cho et al., 2023).

A plausible implication is the general utility of OvO selection as a lightweight, task-agnostic criterion for self-supervised model selection and optimization, with potential applications beyond the exemplified domains.

7. Related Work and Extensions

Key related paradigms include output-vs-ground-truth (OvG) supervision and direct optimization via scoring functions that depend on reference outputs. OvO selection generalizes where ground truth is unavailable or expensive, leveraging robust evaluators or intrinsic confidence metrics. In prompt optimization, SPO extends prior methods such as APE, OPRO, PromptAgent, PromptBreeder, and TextGrad by achieving lower cost and higher efficiency in the absence of references (Xiang et al., 7 Feb 2025). In segmentation, adaptive OvO selection complements approaches based on input or feature fusion, providing a competitive alternative with minimal computational overhead (Cho et al., 2023).

Extensions of OvO selection may incorporate round-robin tournaments, hybrid supervision, and evaluator prompt optimization to further reduce bias and improve global convergence properties.