Randomized Controlled Trials of Service Interventions: The Impact of Capacity Constraints

Abstract: Randomized controlled trials (RCTs), or experiments, are the gold standard for intervention evaluation. However, the main appeal of RCTs, the clean identification of causal effects, can be compromised by interference, when one subject's treatment assignment can influence another subject's behavior or outcomes. In this paper, we formalise and study a type of interference stemming from the operational implementation of a subclass of interventions we term Service Interventions (SIs): interventions that include an on-demand service component provided by a costly and limited resource (e.g., healthcare providers or teachers). We show that in such a system, the capacity constraints induce dependencies across experiment subjects, where an individual may need to wait before receiving the intervention. By modeling these dependencies using a queueing system, we show how increasing the number of subjects without increasing the capacity of the system can lead to a nonlinear decrease in the treatment effect size. This has implications for conventional power analysis and recruitment strategies: increasing the sample size of an RCT without appropriately expanding capacity can decrease the study's power. To address this issue, we propose a method to jointly select the system capacity and number of users using the square root staffing rule from queueing theory. We show how incorporating knowledge of the queueing structure can help an experimenter reduce the amount of capacity and number of subjects required while still maintaining high power. In addition, our analysis of congestion-driven interference provides one concrete mechanism to explain why similar protocols can result in different RCT outcomes and why promising interventions at the RCT stage may not perform well at scale.