A Note on Hypothesis Testing with Random Sample Sizes and its Relationship to Bayes Factors

Authors

  • Oskar Skibskios Author

Keywords:

α-level, Bayes Factor, hypothesis testing, Jeffreys’s paradox

Abstract

Practitioners frequently see frequentist and Bayesian hypothesis testing as "two separate worlds." Although there are theoretical connections, our objective here is to show a real-world scenario in which frequentist hypothesis testing requires caution and to illustrate a practical equivalency between frequentist and Bayesian testing. Specifically, selecting an α level in advance, like 0.05, and applying it to all possible sample sizes is inadmissible if the sample size is random (which is hardly unusual in practical problems where the sample size may be "all available experimental units"). To put it another way, a different overall procedure with the same overall type I error but more power can be found. This alternative process is based on Bayesian testing procedures, which is not by accident.

Downloads

Published

2026-06-29