Measuring population heterogeneity requires heterogeneous populations

Any judgment about population heterogeneity depends on the definition of the sampling frame (1). In a recent paper, Holzmeister et al. (2) (HJBK hereafter) compare different sources of heterogeneity to population heterogeneity. They find that population heterogeneity is much smaller compared to design and analytic heterogeneity as a source of variation in effect sizes. This is important because, if true, it presents an optimistic picture for the generalization of results from one sample to another. However, this claim is puzzling given calls to increase attention to heterogeneity in social science (3). A closer examination of their data and related work (4) (KSJ hereafter) suggests a modification to their conclusion.