(When) Should you adjust inference for multiple hypothesis testing?

(joint with Paul Niehaus and Kaspar Wuthrich) 

Abstract

The use of multiple hypothesis testing adjustments varies widely in applied eco- nomic research, without consensus on when and how it should be done. We provide an economic foundation for this practice. Adjustments are often—but not always— appropriate in our framework when research influences multiple policy decisions. These adjustments depend on the nature of scale economies in the research production function and on economic interactions between policy decisions, with control of classical notions of compound error rates emerging in some but not all cases. Empirical analysis of a unique dataset on research project costs interpreted through the lens of the theory suggests both that MHT adjustments may be warranted and that standard procedures may be too conservative. When research examines multiple outcomes, on the other hand, this motivates aggregating outcomes into sufficient statistics for policy-making.

Paper available at: https://arxiv.org/pdf/2104.13367.pdf