SRank: Functional Overlap Reranking

• Functional Overlap Reranking (SRank) is a suite of post-processing algorithms that reorder candidate outputs by measuring overlaps in salient features such as named entities in QA or functional outputs in code generation.
• The approach operates without retraining the base model by leveraging simple yet effective overlap metrics, applying named-entity intersection in QA and functional agreement across test inputs in code generation.
• Experimental evaluations show that SRank achieves significant performance gains, including a pass@1 improvement of about 6.1% over other rerankers, while enhancing adversarial robustness in natural language and code tasks.

Functional Overlap Reranking (SRank) encompasses a family of model-agnostic post-processing algorithms that reorder candidate outputs—such as answer spans in adversarial question answering or clusters in neural code generation—based on quantified overlaps of salient features or behaviors. Most notably, SRank leverages either named-entity overlap between candidate sentences and questions in natural language QA (Majumder et al., 2021) or functional agreement between output clusters of sampled programs on test inputs in code generation (To et al., 2023). These approaches require no retraining or modification of the underlying generator models and achieve state-of-the-art improvements by exploiting distinct, often shallow, yet highly effective interaction signals among candidates.

1. Formal Definitions of SRank

SRank instantiates specific forms depending on the application domain:

a) Question Answering (QA) (Majumder et al., 2021):

Let $Q$ denote the input question, $C$ the context, and $M$ a span-based QA model. $M$ generates a set $A = \{ (s_1, e_1), ..., (s_n, e_n) \}$ of $n$ candidate answer spans, where each $(s_k, e_k)$ anchors an answer span $e_k$ in sentence $s_k$. For each candidate, a named-entity tagger extracts $NER(Q)$ and $NER(s_k)$. The reranking score is defined as

$$\mathrm{SRank}(Q, (s_k, e_k)) = |NER(s_k) \cap NER(Q)|.$$

Candidates are reranked in descending order of this intersection size; ties are broken by the original model order.

b) Neural Code Generation (To et al., 2023):

Given a naturallanguage programming specification $c$, a code LLM samples $N$ solutions $S = \{s_1, ..., s_N\}$ and $M$ test inputs $Z = \{z_1, ..., z_M\}$ with respective outputs. Solutions are clustered: $C_p$ is the set of code samples whose output vectors on $Z$ are identical. For $K$ clusters $C_1, ..., C_K$, the interaction matrix $I \in \mathbb{R}^{K \times K}$ records functional overlap:

$$I_{ij} = \frac{1}{M} \sum_{k=1}^{M} \mathbf{1}[o_{i,k} = o_{j,k}]$$

where $o_{i,k}$ is the (deterministic) output of all members of $C_i$ on $z_k$. The SRank score for cluster $C_i$ aggregates overlap-weighted cluster features $V_j$:

$R_i = \sum_{j=1}^K I_{ij} V_j.$

The highest-ranked cluster yields the solution returned.

2. Overlap Measures and Algorithms

a) Named-Entity Overlap (QA):

SRank exclusively uses the named-entity intersection:

$\mathrm{overlap}_k = |NER(s_k) \cap NER(Q)|.$

No additional weightings, normalizations, or alternative overlaps (n-gram, TF-IDF, embedding similarity) are used in the evaluated pipeline, though the framework is general (Majumder et al., 2021).

b) Functional Agreement (Code):

Clusters are defined by equivalence on output vectors across test cases. Pairwise overlap is the normalized count of test cases for which two clusters agree:

$$\mathrm{Overlap}(C_i, C_j) = \frac{1}{M} \sum_{k=1}^{M}\mathbf{1}[o_{i,k} = o_{j,k}]$$

leading to an $I$ matrix that models inter-cluster functional similarity.

3. Pipelines and Implementation Details

a) QA Pipeline (Majumder et al., 2021):

• Predict $n$ candidate spans with the base QA model.
• For each, extract the containing sentence and run an NER tagger (AllenNLP’s ELMo-based CoNLL-2003).
• Compute the set intersection $NER(s_k) \cap NER(Q)$ for each candidate.
• Sort candidates by intersection size, then by original model rank.
• Return the highest-ranked span.

Common settings: BiDAF and BERT as base models; beam size $n=10$; max input 400 tokens; per-example reranking adds $\mathcal{O}(n L)$ NER overhead, negligible versus the model’s forward pass.

b) Code Generation Pipeline (To et al., 2023):

• Sample $N$ solutions and $M$ test inputs (e.g., $N=M=100$).
• Execute all $s_i$ on all $z_k$ to get output vectors.
• Cluster solutions by identical output vectors across $Z$.
• Construct the $K \times K$ interaction matrix $I$ of overlaps.
• Define $V_j$ as cluster feature (e.g., cluster size or pass rate).
• Calculate $R_i = \sum_{j=1}^K I_{ij} V_j$, sort clusters, and extract a representative solution from the winner.

Computational cost is dominated by $O(NM)$ executions; $K \ll N$ in most cases.

4. Experimental Evaluation and Comparative Results

Table 1: pass@1 (%) on HumanEval and MBPP-S (To et al., 2023)

Model	Greedy	CodeT	Coder-Reviewer	SRank
WizardCoder34B	68.90	72.36	–	75.31
CodeGen2.5-Instr.	28.05	56.81	45.63	60.55
StarCoder	39.63	50.51	38.71	53.99
Codex002	47.00	65.80	66.90	69.66

SRank achieves average pass@1 improvement of ≈6.1% over CodeT and Coder-Reviewer rerankers.

QA Results (Majumder et al., 2021):

• On Adversarial SQuAD (AddSent), BiDAF+SRank achieves 45.4 F1 / 38.0 EM (vs. vanilla BiDAF, 21.4 / 16.0, and BiDAF+SLN, 22.8 / 17.2).
• For BERT, gains are similar (BERT+SRank: 61.2/53.6 vs. BERT+QAInfoMax, 41.8/37.2).
• On clean data, SRank-only BERT can experience minor drops due to answer type mismatch but yields strong net improvements in adversarial settings.

5. Analysis, Robustness, and Practical Trade-offs

a) Adversarial Robustness (QA):

SRank exploits that adversarial distractors in context rarely share named entities with the question, allowing span filtering without any retraining. This model-agnostic and post-hoc property makes it applicable to any span-predicting reader.

b) Robustness (Code):

Functional overlap amplifies the consensus among correct solutions: genuinely correct clusters tend to agree on outputs, while “buggy” clusters disagree. Especially under small $N$ and $M$, this consensus effect outperforms single-cluster features and reduces noise in cluster scoring.

c) Limitations:

• In QA, SRank may fail if all candidate sentences with shared entities lack the correct answer or if entity overlap does not imply answer type match.
• In code, the approach assumes perfect clustering by exact outputs; richer semantic (canonical) clustering may further improve effectiveness.
• Results are presented for English QA and Python code only; generalization to other languages or domains is not yet demonstrated.

6. Extensions, Limitations, and Future Perspectives

SRank’s design is agnostic to the underlying overlap metric and to the cluster features. While only named-entity and functional overlap are employed in the respective evaluated domains, the general framework could incorporate richer content-match (n-grams, TF-IDF, embeddings), additional verifiers, or LLM log-probs as cluster features. Scaling SRank to massive candidate sets may require approximations or subset sampling. Its strong empirical performance underscores both the potential of consensus-based reranking and possible vulnerabilities in current adversarial benchmarks, motivating future development of defenses premised on deeper semantic or pragmatic reasoning beyond entity or functional overlap (Majumder et al., 2021, To et al., 2023).