Learning from (Failed) Replications: Cognitive Load Manipulations and Charitable Giving

Replication of empirical studies is much more than a tool to police the field. Failed replications force us to recognize that seemingly arbitrary design features may impact results in important ways. We describe a study that used a cognitive load manipulation to investigate the role of the deliberative system in charitable giving and a set of failed replications of that study. While the original study showed large and statistically significant results, we failed to replicate using the same protocol and the same subject pool. After the first failed replication, we hypothesized that the order our study was taken in a set of unrelated studies in a laboratory session generated the differences in effects. Three more replication attempts supported this hypothesis. The study demonstrates the importance of replication in advancing our understanding of the mechanisms driving a particular result and it questions the robustness of results established by cognitive load tests.