Does Excluding Some Groups From Research Designs Improve Statistical Power?

This article aims to contribute to a controversy over whether excluding some small or incomplete groups from a sample improves statistical power in group research designs (designs that relate group-level characteristics to group-level outcome measures). In a series of simulation studies, we examined the trade-off between lower reliability and smaller sample size that occurs when very small groups, or incomplete groups are excluded. Distinguishing reflective aggregation models (where scores for different group members are interchangeable) and formative aggregation models (where scores for different group members are not interchangeable), we analyzed the impact that the number of groups, the number of individuals within groups, intraclass correlation (ICC[1]) values, and interrater agreement have on statistical power. The results provided evidence that excluding groups is mostly ill-advised and may fail to improve the conclusions that researchers draw from their results. Common practice and the assumptions that researchers make when excluding groups from their samples are discussed.