Design and Analysis of Switchback Experiments
Abstract
In switchback experiments, a firm sequentially exposes an experimental unit to a random treatment, measures its response, and repeats the procedure for several periods to quantify the treatment effect. Although practitioners have widely adopted this experimental design technique, the development of its theoretical properties and the derivation of optimal designs have been elusive. In this paper, we establish the necessary results for practitioners to apply this powerful class of experiments with minimal assumptions. Our main result is the derivation of the optimal design of switchback experiments under a range of different assumptions on the order of the carryover effect, that is, the length of time a treatment persists in impacting the outcome. We cast the optimal experimental design problem as a minimax discrete optimization problem, identify the worst-case adversarial strategy, establish structural results, and solve the reduced problem via discrete convexity. For switchback experiments conducted under the optimal design, we provide two approaches for performing inference. The first provides exact randomization based p-values, and the second uses a new finite population central limit theorem to conduct conservative hypothesis tests and build confidence intervals. We further provide theoretical results when the order of the carryover effect is misspecified. For firms that possess the capability to run multiple switchback experiments, we also provide a data-driven procedure to identify the likely order of the carryover effect. We conduct extensive simulations to study the empirical properties of our results, and conclude with some practical suggestions. This talk is based on joint work with Iavor Bojinov and David Simchi-Levi, and in collaboration with Zalando, a European fashion retailer.
