Prompt Stability Score Metrics

• Prompt Stability Score (PSS) is a metric that measures the consistency of LLM outputs against perturbations such as stochastic sampling, prompt paraphrasing, and multi-agent task chaining.
• Various methodologies, including cosine similarity of semantic embeddings, Krippendorff’s alpha, flip rate, and AUC-E, are employed to capture and diagnose output variability and unpredictability.
• PSS is pivotal for model selection, deployment safety, and trustworthiness by highlighting vulnerabilities in LLM behavior when subtle changes occur in prompt wording or execution conditions.

Prompt Stability Score (PSS) is a class of metrics quantifying the consistency of LLM outputs—across repeated executions, stochastic decoding, or semantically equivalent prompt paraphrasing. Its core purpose is to diagnose the vulnerability of LLM-driven systems to unpredictable or incoherent behavior induced by subtle changes in prompt wording or repeated sampling, especially in zero-shot or structured workflows. Approaches to PSS span continuous/semantic agreement metrics, inter/intra-coder reliability measures, paraphrase-induced flip rates, and area-under-curve summaries of output invariance. Prompt stability is recognized as a distinct and orthogonal dimension of LLM evaluation, of foundational importance to model selection, trustworthiness, multi-agent orchestration, and deployment safety (Ma et al., 17 Sep 2025, &&&1&&&, Barrie et al., 2024, Kolbeinsson et al., 29 Jan 2026).

1. Definitions and Core Concepts

Prompt Stability Score operationalizes the degree to which a model’s outputs remain invariant, or meaningfully consistent, under three principal perturbation classes:

• Sampling stochasticity: For a fixed prompt, repeated LLM executions (with nonzero temperature, top-k, or nucleus sampling) produce a distribution of outputs. PSS quantifies whether these outputs are semantically or functionally similar (Chen et al., 19 May 2025, Barrie et al., 2024).
• Prompt paraphrasing: For a baseline task prompt $P_0$ and a set of semantically equivalent variants $P_1,\dots,P_K$ (generated by paraphrasing, templating, or style transfer), PSS assesses sensitivity of predictions to these changes (Ma et al., 17 Sep 2025, Kolbeinsson et al., 29 Jan 2026, Barrie et al., 2024).
• Task context or agent chaining: In LLM pipelines (e.g., multi-agent planners), prompt inconsistency can propagate through aggregation and summarization layers, amplifying instability (Chen et al., 19 May 2025).

The essence of PSS is to collapse potentially high-dimensional output variability into a single interpretable metric reflecting reliability, replicability, and downstream trustworthiness.

2. Measurement Methodologies

Measurement strategies for PSS are domain- and application-dependent, but cluster into several technical frameworks:

A. Semantic Stability (Cosine Similarity-Based)

Given a prompt $p$, generate $N$ independent LLM outputs $\{y_1,\dots,y_N\}$, typically under stochastic decoding. Each output is embedded into a semantic vector $v_i = \varphi(y_i)$ (e.g., Sentence-BERT). The pairwise cosine distances $d_{ij} = 1 - (v_i \cdot v_j) / (\|v_i\|\|v_j\|)$ are averaged:

$$S(p) = 1 - \frac{2}{N(N-1)} \sum_{1 \leq i < j \leq N} d_{ij} = \frac{2}{N(N-1)} \sum_{i<j} \frac{v_i \cdot v_j}{\|v_i\|\|v_j\|}$$

$S(p) \in [0,1]$, with higher values indicating greater semantic stability. This is used as Prompt Stability Score in general-purpose and multi-agent settings (Chen et al., 19 May 2025).

B. Inter/Intra-Prompt Reliability (Krippendorff’s Alpha)

• Intra-PSS: Repeated model runs at fixed prompt, treated as independent coders. Krippendorff’s $\alpha$ is computed across an $n \times R$ labeling matrix; average over iterations gives $PSS_{intra}$.
• Inter-PSS: Fix example, vary across prompt paraphrases; compute $\alpha$ across $n \times m$ matrices for different temperatures or variant intensities. Mean $\alpha$ across settings yields $PSS_{inter}$.
• Aggregate PSS: $PSS = \frac{1}{2}(PSS_{intra} + PSS_{inter})$ (Barrie et al., 2024).

C. Flip Rate and Instance-Level PSS

For classification, anchor-based flip metrics quantify the fraction of paraphrases that alter the predicted label relative to a base prompt:

$$PSS(x) = 1 - \frac{1}{K} \sum_{i=1}^K 1[f(P_i, x) \ne f(P_0, x)]$$

Dataset-level stability is then $S(P) = \frac{1}{N} \sum_x PSS(x)$. The symmetric variant averages over all prompt pairs. Lower flip rates indicate higher prompt stability (Kolbeinsson et al., 29 Jan 2026).

D. Elasticity Curve and Area Under Curve (AUC-E)

In code generation, PromptSE and PromptSELight define correctness scores over paraphrased prompts and aggregate stability as an “elasticity” curve over perturbation levels $d$:

$$\mathrm{Elasticity}(p,d) = 1 - \frac{1}{|V^p_d|} \sum_{v \in V^p_d} |\mathrm{Score}(p) - \mathrm{Score}(v)|$$

AUC-E, the area under $\mathcal E(d)$ for several $d$, summarizes the model’s invariance and is used as the empirical Prompt Stability Score:

$PSS = \mathrm{AUC}\text{-}\mathrm{E}$

Values near $1$ correspond to stable models under expression/style variation (Ma et al., 17 Sep 2025).

3. Practical Algorithms and Protocols

Protocols are tuned for computational feasibility and scientific reproducibility.

• For semantic stability (S(p)): Draw $N$ LLM samples at fixed hyperparameters, compute embeddings, and average cosine similarities (Chen et al., 19 May 2025).
• Krippendorff’s $\alpha$ PSS: Run $R \gg 1$ repeat samples for intra-PSS, or paraphrase and label with single samples for inter-PSS, then compute alpha per (Barrie et al., 2024).
• Elasticity/AUC-E (Code LLMs): For each prompt and each perturbation level $d$, collect sample outputs, evaluate correctness (possibly with probability weighting), compute elasticity, and aggregate over $d$ using Simpson’s rule:

$$\mathrm{AUC}\text{-}\mathrm{E} = \frac{1}{9} [\mathcal{E}(0.1) + 4 \mathcal{E}(0.2) + \mathcal{E}(0.3)]$$

(Ma et al., 17 Sep 2025).

• Flip-rate PSS (Clinical): For each datapoint, compute the label under base and $K$ paraphrased prompts, aggregate the per-example stability into a global score (Kolbeinsson et al., 29 Jan 2026).

Most protocols recommend reporting both intra- and inter-prompt (or paraphrase) stability, providing empirical thresholds (e.g., PSS $< 0.8$ suggesting critical instability), and controlling for extraneous randomness by fixing all non-prompt hyperparameters (Barrie et al., 2024).

4. Empirical Findings and Comparative Analysis

Empirical investigations across domains have revealed the following patterns:

• Stability and performance are decoupled: For code LLMs, peak performance (e.g., Pass@1) and AUC-E-based PSS span nearly orthogonal dimensions; high-accuracy models can be unstable under paraphrase, and vice versa (Ma et al., 17 Sep 2025, Kolbeinsson et al., 29 Jan 2026).
• Model scale and architecture effects: Smaller models have sometimes demonstrated higher prompt stability than larger models, undermining scale-invariance assumptions (Ma et al., 17 Sep 2025).
• Flip-rate optimization in clinical LLMs: Jointly optimizing for accuracy and prompt stability (e.g., via a combined loss with both terms) can markedly reduce flip rates—with only moderate or negligible accuracy loss in most settings (Kolbeinsson et al., 29 Jan 2026).
• Semantic PSS matches execution stability: Embedding-based semantic agreement correlates strongly with empirical task consistency and execution success; models with higher $S(p)$ yield lower agent-level variance and more robust system aggregation (Chen et al., 19 May 2025).
• Reliability limits replicability: Low PSS values correspond to settings where output variability impedes scientific conclusions or application reliability, e.g., in text annotation pipelines (Barrie et al., 2024).
• Proxy and approximator utility: Binary variants and light-weight flip-rate measures often correlate well with more expensive, fully semantic or probabilistic metrics (Pearson $r \sim 0.7$–$0.75$) (Ma et al., 17 Sep 2025, Kolbeinsson et al., 29 Jan 2026).

5. Theoretical Significance and Systemic Implications

Prompt stability exerts profound influence on both micro- and macro-level LLM system reliability.

• Pipeline propagation: In agentic or compositional workflows, small prompt-level variances induce aggregate deviations, provably bounded as:

$$P(|s - \hat{s}| \geq \epsilon) \leq 2 \exp\left( -\frac{\epsilon^2}{2 \sum_i (u_i v_i)^2 \mathrm{Var}(x_i)} \right)$$

with $\mathrm{Var}(x_i) \propto 1 - S(p_i)$, establishing high prompt stability as a necessary condition for robust system-level execution (Chen et al., 19 May 2025).

• Complementarity with calibration and uncertainty: Flip-rate-based instability only partially overlaps with standard measures of predictive uncertainty or conformal set size, necessitating explicit PSS evaluation (Kolbeinsson et al., 29 Jan 2026).
• Composite scoring: PSS can be fused with accuracy metrics for composite model selection, such as $$\mathrm{PSS}_{\alpha,\beta} = \alpha \cdot \mathrm{Pass@1} + \beta \cdot \mathrm{AUC}\text{-}\mathrm{E}$$, calibrating for both peak performance and real-world robustness (Ma et al., 17 Sep 2025).

6. Best Practices, Limitations, and Future Directions

• Diagnostic integration: PSS should be computed routinely for any foundational prompt or annotation pipeline. Instability (e.g., PSS $< 0.8$) requires prompt revision, clarification, or operational modality change (e.g., few-shot over zero-shot) (Barrie et al., 2024).
• Transparency and reproducibility: All baseline prompts, paraphrase sets, code, and stability metrics should be documented and shared (Barrie et al., 2024).
• Embedding/model dependency: Embedding models (e.g., Sentence-BERT, Universal Sentence Encoder) are a critical source of bias in semantic PSS (Chen et al., 19 May 2025).
• Sampling overhead: Stability computation can be expensive; learned evaluator models (e.g., fine-tuned LLaMA regressors) or binary proxies ameliorate this at some fidelity cost (Chen et al., 19 May 2025, Ma et al., 17 Sep 2025).
• Task- and agent-adaptive thresholds: Fixed stability thresholds may miss context-dependency. Dynamic adaptations are an open direction (Chen et al., 19 May 2025).
• Alignment with correctness and safety: High PSS does not guarantee factual accuracy or safety; further integration of correctness constraints is needed (Chen et al., 19 May 2025).

7. Comparative Table of PSS Methodologies

Reference	PSS Definition / Metric	Core Application Domain
(Chen et al., 19 May 2025)	Mean pairwise semantic similarity $S(p)$	Multi-agent LLM, general-purpose execution
(Ma et al., 17 Sep 2025)	Area under elasticity curve (AUC-E)	Code-generation LLMs, prompt paraphrasing
(Barrie et al., 2024)	Krippendorff’s $\alpha$ (intra/inter)	Text annotation/classification pipelines
(Kolbeinsson et al., 29 Jan 2026)	Anchor-based flip rate/PSS $(1-\mathrm{flip\ rate})$	Clinical LLM classification

Each methodology prescribes context-sensitive protocols but is united by the aim to produce a scalar, interpretable estimate of output consistency under model or prompt variance, foundational to the scientific and applied evaluation of LLM-based systems.