Abstract
We consider experimentation in settings where, due to interference or other concerns, experimental units are coarse. ‘Region-split’ experiments on online platforms, where an intervention is applied to a single region over some experimental horizon, are one example of such a setting. Synthetic control is the state-of-the-art approach to inference in such experiments. The cost of these experiments is high since the opportunity cost of a sub-optimal intervention is borne by an entire region over the length of the experiment. More seriously, correct inference requires assumptions limiting the ‘non-stationarity’ of test and control units that we demonstrate fail in practice. So motivated, we propose a new adaptive approach to experimentation, dubbed Synthetically Controlled Thompson Sampling (SCTS). SCTS is guaranteed to identify the optimal treatment without the attendant non-stationarity assumptions of the status quo, thereby allowing for robust inference. In addition, SCTS minimizes the cost of experimentation by incurring near-optimal, square-root regret in the experimental horizon, as opposed to linear regret for the status quo. Experiments on synthetic and real world data highlight the relative merits of SCTS in regard to both the cost of experimentation and the robustness of inference.