RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization

Abstract: Serving diverse NLP workloads with LLMs is costly: at one enterprise partner, inference costs exceeded $200K/month despite over 70% of queries being routine tasks well within the capability of smaller models. We present RouteNLP, a closed-loop framework that routes queries across a tiered model portfolio to minimize cost while satisfying per-task quality constraints. The framework integrates three components: a difficulty-aware router with shared task-conditioned representations trained on preference data and quality signals; confidence-calibrated cascading that uses conformal prediction for distribution-free threshold initialization; and a distillation-routing co-optimization loop that clusters escalation failures, applies targeted knowledge distillation to cheaper models, and automatically retrains the router, yielding over twice the cost improvement of untargeted distillation. In an 8-week pilot deployment processing ~5K queries/day at an enterprise customer-service division, RouteNLP reduced inference costs by 58% while maintaining 91% response acceptance and reducing p99 latency from 1,847 ms to 387 ms. On a six-task benchmark spanning finance, customer service, and legal domains, the framework achieves 40-85% cost reduction while retaining 96-100% quality on structured tasks and 96-98% on generation tasks, with human evaluation confirming that 74.5% of routed generation outputs match or exceed frontier-model quality.

• The paper presents a novel framework that unifies difficulty-aware routing, conformal cascading, and closed-loop distillation to reduce inference costs by up to 85% while meeting quality SLAs.
• It employs a lightweight DistilBERT-based multi-task router with confidence-calibrated cascading, which dynamically routes queries based on task difficulty and uncertainty estimates.
• Empirical evaluations across diverse NLP tasks demonstrate significant cost savings, robust quality retention, and effective pilot deployment in production settings.

RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization

Problem Context and Motivation

LLM deployment in production environments is constrained by high inference costs, latency SLAs, and heterogeneous workload difficulty. Empirical evidence from enterprise settings shows that the majority of queries are structurally simple and do not require the use of high-cost, frontier LLMs. However, existing model routing and cascading techniques are generally evaluated in academic or single-benchmark settings, lack explicit SLA calibration, and treat the set of candidate models as static. The need is acute for frameworks that adaptively select model tiers to minimize cost while automatically improving the capacity of lower tiers in response to observed failures, all under strict quality constraints.

RouteNLP Framework

The RouteNLP framework addresses these gaps by unifying four system components: a difficulty-aware, multi-task router; confidence-calibrated cascading using conformal prediction; a distillation–routing co-optimization loop; and a multi-domain SLA-constrained evaluation methodology.

Formulation

Given a portfolio $\mathcal{M} = \{m_1, \ldots, m_K\}$ ordered by cost, RouteNLP learns a routing policy $r_\theta$ that, for a query $x$ of task type $t$, minimizes expected cumulative inference cost while ensuring per-task quality meets required thresholds $\tau_t$:

$$\min_{r_\theta} \; \mathbb{E}_x\big[\sum_{k=1}^{k^*(x)} c_{k,t}\big] \;\; \text{s.t.} \;\; \mathbb{E}_x[q(m_{k^*(x)},x)] \ge \tau_t$$

Difficulty-Aware Router

The router is realized as a lightweight DistilBERT encoder with a multi-task classification head, incorporating learned task embeddings concatenated to the [CLS] token. The router is jointly optimized on per-task quality labels, preference data from pairwise comparisons, and a composite loss reflecting both cost minimization and SLA-constrained quality. Training labels are extracted by exhaustive all-model evaluation and the router predicts the cheapest model sufficient for the query's quality constraint. Multi-task conditioning leads to a single encoder that generalizes across domains and task types.

Confidence-Calibrated Cascading

For each routed query, RouteNLP applies conformal prediction to token-level uncertainty estimates, following the distribution-free risk control methodology. Uncertainty statistics are calibrated over a held-out validation set to determine per-tier, per-task escalation thresholds. Escalation across model tiers continues until confidence meets the desired level or until the most capable model is reached. Conformal guarantees are marginal and monitored for violations under distribution shift; threshold calibration is recomputed in response to drift.

Distillation–Routing Co-Optimization

Distinct from prior work, RouteNLP integrates a closed feedback loop whereby escalation logs are clustered via a PCA-reduced embedding of the router's penultimate layer, followed by $k$-means clustering. Clusters with high density and large average quality gaps define query cohorts for targeted distillation using sequence-level KD from the highest-tier model. Iterative fine-tuning of lower-tier models with this targeted data is immediately followed by router retraining and threshold recalibration. This loop converges in 2–3 iterations and demonstrably shifts query allocation away from expensive models.

Empirical Evaluation

Benchmark and Pilot Setup

Evaluation spans six production-like NLP tasks (financial NER and summarization, customer service intent and response, and legal clause extraction and risk assessment) using enterprise-annotated public datasets and a production pilot with 5K queries/day over 8 weeks in a customer service environment. The model portfolio includes fine-tuned DistilBERT (T1), Mistral-7B (T2), Mixtral-8x7B (T3), and GPT-4-Turbo (T4), covering three orders of magnitude in cost.

Baselines include Always-T4 (upper bound), RouteLLM, Hybrid LLM, AutoMix, and FrugalGPT, with 2-to-4 tier adaptations for fairness.

Main Results

RouteNLP achieves 40–85% cost reduction while retaining 96–100% quality on structured tasks and 96–98% on generation tasks. Compared to the best prior router RouteLLM, RouteNLP reduces average cost from 0.246 to 0.159 (relative to Always-T4), a statistically significant result ($p < 0.001$), and reduces SLA violation rates from 17.2% (RouteLLM) to 2.3%. Human evaluation shows that 74.5% of generated outputs are rated as matching or exceeding frontier-model quality, though 8–9% of all responses show substantial degradation.

Ablation reveals that the closed-loop co-optimization doubles the cost reduction versus random or untargeted distillation (21.7% vs. 9.4% decrease in cost ratio at equal data volume) and that multi-task conditioning and conformal cascading are each essential for joint cost–quality optimization.

Per-Task Performance

Structured extraction and classification tasks route 68–72% of queries to T1 and attain $>99\%$ quality retention vs. T4. Generation tasks benefit less from aggressive routing but achieve substantial savings (e.g., financial summarization: 47% cost saving at 96% ROUGE-L retention; customer service response: 42% cost saving at 96% BERTScore retention).

System Robustness

RouteNLP is robust to modest distribution and quality threshold shifts, with graceful degradation in cost–quality tradeoff. Coverage violations under domain shift can exceed the 5% target, emphasizing the necessity of recalibration and production monitoring. The router's reliance on BERTScore as a proxy remains reliable in-distribution ($\sim$85% agreement with human judgment), but further validation is required under distribution changes.

Pilot Deployment

In production pilot, RouteNLP reduced inference cost by 58% (within 4 percentage points of the simulation), maintained 91% response acceptance, and cut p99 latency from 1,847 ms to 387 ms. Routing shares and cost ratios closely tracked simulation predictions. Analysis of pilot escalation logs uncovered additional live failure modes (OCR errors, conversational context dependencies) not captured in benchmarks.

Retrospective domain expert audits confirmed rates of unacceptable responses were acceptably low (2.6%, +0.8pp over baseline) and qualitative performance was stable over dynamic portfolio changes and temporary outages.

Implications and Future Directions

Practically, RouteNLP demonstrates the feasibility of rigorously optimized tiered serving for enterprise LLM workloads across domains. Strong empirical cost–quality tradeoffs and resilience to distributional perturbations establish RouteNLP as an operational framework for production environments with heterogeneous difficulty and stringent constraints.

Theoretically, the integration of targeted system-level distillation as a portfolio improvement mechanism moves beyond static routing/cascading, closing the loop between observed failures and the serving policy. The use of conformal prediction for SLA-aware calibration, while only affording marginal guarantees, is scalable and production-viable given suitable monitoring.

Potential extensions include:

• Application to multi-turn, agentic, or document-level workflows where query difficulty is non-local,
• Online recalibration strategies for highly dynamic environments,
• Automatic fairness and demographic balancing in routing decisions,
• Extended support for non-English and more diverse task types,
• Investigation of richer meta-learning or reinforcement learning for router and portfolio adaptation.

Conclusion

RouteNLP presents a unified, operationally-focused framework for cost-efficient, quality-constrained LLM serving, integrating multi-task routing, conformal risk control, and targeted system-level distillation. Strong numerical cost savings, thorough pilot validation, and systematic ablation establish its value for enterprise-grade NLP workloads. While current guarantees are marginal and depend on robust distribution calibration, the approach provides a blueprint for scalable, adaptive LLM serving under realistic constraints and evolving production requirements (2604.23577).