July 15, 2026
In the literature, there has been conflicting guidance about when to use multiple testing correction or adjustment. Also, the distinction between a confirmatory vs an exploratory analyses can make all the difference. The guidance has usually been that it was only needed for confirmatory studies, but others argue that is it especially important in unplanned experiments. Other authors have also suggested the distinction should be dependent on the type of outcome variables considered or on the way in which the multiple hypotheses are combined (disjunction vs conjunction test; Dmitrienko and D’Agostino, 2013; Rubin 2021). Another concern was about whether a family of tests belong to one experiment. The authors have, therefore, come up with a unifying, guiding principle to assist both statisticians and applied researcher to decide whether to adjust for multiple testing. In this paper, they focused on the Bonferroni correction in their examples but any of the other possible multiple correction methods could be instead employed.
Their guiding principle revolves around “the decision to adjust for multiple testing should be inherently linked to how the significance of results affects the reporting and interpretation of the findings. The general principle is that multiple testing should be adjusted for if and only if authors, when reporting and interpreting their findings, put more emphasis on the results of one or several of the tests because of their small p-value(s) (cf. Boulesteix and Hoffmann, 2024).” Therefore, this guiding principle was first sketches by Boulesteix and Hoffmann (2024) and then more detail added by the authors. As they have suggested, it can be quite useful in design, analysis, reporting, and interpretation of studies. They have said that their guiding principle is unifying in the sense that it is not a complex set of rules which depend on the type of study.
They laid out five simple cases like primary endpoints in an RCT to multiplicity of analyses strategies in Table 2 and gave a common recommendation and the guiding principle recommendation. For an RCT with multiple primary endpoints that is declared successful as soon as at least one of the tests is significant, according to the principle, one should adjust for multiple testing. In contract, the results of the tests for secondary endpoints of an RCT are usually reported on equal footing so according to the guiding principle, should not adjust for multiple testing. In general, when authors perform many different analysis strategies and then selectively report the results of those yielding the most significant results in the paper, they put more emphasis on the most significant results when writing their manuscript. Then according to the principle, the adjustment is necessary.
As the authors have stated, this principle has provided a framework to guide researchers on when to adjust and, if so, over which set of tests, beyond trivial situations. It does not make this decision easy or self-evident, but certainly less difficult. They have stated that this principle is unifying in the sense that it subsumes several widely accepted recommendations as special cases. They demonstrated its application in three case studies. Their guiding principle also makes clear, beyond the context of adjustment for multiple testing in a narrow sense, that publication bias should be adjusted for when conducting meta-analyses. This is because the scientific literature has placed greater emphasis on significant rather than non-significant results. If researchers try to select significant results selectively, they can try to show both unadjusted and adjusted p-values in their presentation. One of the items they missed was reporting of the applied correction itself and other guidelines like STROBE for observational studies (Vandenbrouckel et al, 2007) contain no guidance while SPIRIT (Chan et al, 2025) and CONSORT for randomized studies (Hopewell et al, 2025) are vague and do not make firm recommendations for multiple testing. Essentially, the guidance principle does not resolve the many misunderstandings and problems (Goodman, 2008; Greenland et al, 2016)that arise from use of statistical significance in the medical literature (Amrhein et al, 2019, Wasserstein et al, 2019). The guidance is not meant to defend significance testing but just provide guidance.
Usha Govindarajulu
Keywords: multiple testing, adjustment, Bonferroni correction, guiding principle, p-values
References:
Amrhein, V., S. Greenland, and B. McShane. 2019. “Scientists Rise Up Against Statistical Significance.” Nature 567, no. 7748: 305–307.
Boulesteix, A.-L., and S. Hoffmann. 2024. “To Adjust or Not to Adjust: It Is Not the Tests Performed That Count, But How They Are Reported and Interpreted.” BMJ Medicine 3, no. 1: e000783.
Chan, A.-W., I. Boutron, S. Hopewell, et al. 2025. “SPIRIT 2025 Statement: Updated Guideline for Protocols of Randomised Trials.” Lancet 405, no. 10491: e19–e27.
Dmitrienko, A., and R. D’Agostino. 2013. “Traditional Multiplicity Adjustment Methods in Clinical Trials.” Statistics in Medicine 32, no. 29: 5172–5218.
Goodman, S. 2008. “ A Dirty Dozen: Twelve p-Value Misconceptions.” In Seminars in Hematology, Vol. 45, 135–140. Elsevier.
Greenland, S., S. J. Senn, K. J. Rothman, et al. 2016. “Statistical Tests, P Values, Confidence Intervals, and Power: A Guide to Misinterpretations.” European Journal of Epidemiology 31, no. 4: 337–350.
Hoffman S, Lemster S, Collins G, Hapfelmeier A, Heinze G, Mayr A, Schmid M, Wilcke JC, and Boulesteix A-L (2026). “When to Adjust for Multiple Testing: A Unifying Guiding Principle” Biometrical Journal
https://doi.org/10.1002/bimj.70148
Rubin, M. 2021. “When to Adjust Alpha During Multiple Testing: A Consideration of Disjunction, Conjunction, and Individual Testing.” Synthese 199, no. 3: 10969–11000.
Vandenbrouckel, J. P., E. von Elm, D. G. Altman, et al. 2007. “Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration.” PLoS Medicine 4, no. 10: 1628–1655.
Wasserstein, R. L., A. L. Schirm, and N. A. Lazar. 2019. “Moving to a World Beyond p < 0.05.” American Statistician 73, no. sup1: 1–19.
https://onlinelibrary.wiley.com/cms/asset/6c0ce1eb-ba2f-4c96-9edc-35c5943971c9/bimj70148-fig-0001-m.jpg