PLanet: Formalizing Experimental Design

Abstract: Carefully constructed experimental designs are essential for drawing valid, generalizable conclusions from scientific studies. Unfortunately, experimental design plans can be difficult to specify, communicate clearly, and relate to alternatives. In response, we introduce a grammar of experimental design that provides composable operators for constructing assignment procedures (e.g., Latin square). We implement this grammar in PLanet, a domain-specific language (DSL) that constructs assignment plans in three stages: experimental unit specification, trial-order construction, and order-to-unit mapping. We evaluate PLanet's expressivity by taking a purposive sample of recent CHI and UIST publications, representing their experiments as programs in PLanet, and identifying ambiguities and alternatives. In our evaluation, PLanet could express 11 out of 12 experiments found in sampled papers. Additionally, we found that PLanet constructs helped make complex design choices explicit when the researchers omit technical language describing their study designs.