Singhing with Confidence: Visualising the Performance of Confidence Structures

Abstract: Confidence intervals are an established means of portraying uncertainty about an inferred parameter and can be generated through the use of confidence distributions. For a confidence distribution to be ideal, it must maintain frequentist coverage of the true parameter. This can be represented for a precise distribution by adherence to a cumulative unit uniform distribution, referred to here as a Singh plot. This manuscript extends this to imprecise confidence structures with bounds around the uniform distribution, and describes how deviations convey information regarding the characteristics of confidence structures designed for inference and prediction. This quick visual representation, in a manner similar to ROC curves, aids the development of robust structures and methods that make use of confidence. A demonstration of the utility of Singh plots is provided with an assessment of the coverage of the ProUCL Chebyshev upper confidence limit estimator for the mean of an unknown distribution.