Principled Tuning of the DS-CP Regularization Parameter

Develop principled strategies for tuning the regularization parameter lambda in Domain-Shift-Aware Conformal Prediction, which replaces the test-point weight in the empirical score distribution, to balance coverage validity and prediction-set efficiency under domain shift.

Background

The DS-CP framework mitigates instability from extreme test-point weights by introducing a regularization parameter lambda that depends only on calibration data. Setting lambda to 1 yields a natural baseline and recovers standard conformal prediction in the exchangeable case.

The authors note that the choice of lambda involves a trade-off between validity and efficiency. While a fixed default is practical, identifying more principled tuning strategies is explicitly stated as an open problem, indicating the need for methods that adapt lambda to domain shift while preserving coverage guarantees.

References

Second, while the proposed regularization step prevents degenerate prediction sets, the choice of the regularization parameter involves a trade-off between validity and efficiency. Although setting it to 1 provides a natural and fair baseline, more principled tuning strategies remain an open problem.

— Domain-Shift-Aware Conformal Prediction for Large Language Models  (2510.05566 - Lin et al., 7 Oct 2025) in Section 6 (Discussion)