Simulation and inference on purely observational methods of monitoring vaccine effectiveness post-deployment: none is reliable without precise information on population behaviour

Abstract: Two observational methods are currently being used to monitor post-deployment vaccine effectiveness: the obvious crude method comparing rate testing positive per head of vaccinated population with that rate per head of unvaccinated population; and the test-negative case control (TNCC) method. The two methods give very different results. We want to know whether either method is reliable. We assume either a homogeneous population or one partitioned into two homogeneous subsets which differ only in their not-directly-observable healthcare-seeking behaviour including probability of getting vaccinated. We first consider uniform independent priors on the probabilities of being hospitalised conditional on subset, vaccination status, and infection status. We simulate from the resulting model and observe the TNCC estimate, the crude estimate, and the Bayesian central 95% confidence interval on vaccine effectiveness represented as log ratio of odds ratios for infection with and without vaccination. With these wide open priors, even when the population is homogeneous, the Bayesian 95% confidence interval typically has a width of nearly 4 nats (55-fold), implying too much uncertainty for the data collected to be of any use in monitoring effectiveness. There do exist some tight priors under which the data is useful: some lead to TNCC being more accurate while with others the crude estimate is more accurate. Thus using only data from those spontaneously choosing to be tested, we find that neither method is reliably better than the other, and indeed that the desired information is not present in this data. We conclude that effective monitoring of vaccine effectiveness and side-effects requires either strong information on the population's behaviour, or ongoing randomised controlled trials (RCTs), rather than just choosing whichever of TNCC and crude estimate gives the result we prefer to find.