This chapter presents hypothesis testing which is used to evaluate whether the available data provide sufficient evidence to support a certain hypothesis. The main idea is to play devil's advocate and assume a null hypothesis, which contradicts our hypothesis of interest. We explain how to use parametric modeling to implement this idea, and define the p-value. We prove that thresholding the p-value controls the probability of false positives. In addition, we define the power of a test, which quantifies the test's ability to identify positive findings. Next, we show how to perform hypothesis testing without a parametric model, focusing on the permutation test. Then, we discuss multiple testing, a setting where many tests are performed simultaneously. Finally, we provide three reasons why hypothesis testing should not be used as the only stamp of approval for scientific discoveries. First, hypothesis testing does not necessarily identify causal effects; it is complementary to causal inference. Second, small p-values do not imply practical significance. Third, relying on p-values to validate findings produces a strong incentive to cherry-pick results.

All the NHSTs in previous chapters compare two dependent variable means. When there are three or more group means, it is possible to use unpaired two-sample t-tests for each pair of group means, but there are two problems with this strategy. First, as the number of groups increases, the number of t-tests required increases faster. Second, the risk of Type I error increases with each additional t-test.

The analysis of variance (ANOVA) fixes both problems. Its null hypothesis is that all group means are equal. ANOVA follows the same eight steps as other NHST procedures. ANOVA produces an effect size, η2. The η2 effect size can be interpreted in two ways. First, η2 quantifies the percentage of dependent variable variance that is shared with the independent variable’s variance. Second, η2 measures how much better the group mean functions as a predicted score when compared to the grand mean.

ANOVA only says whether a difference exists – not which means differ from other means. To determine this, a post hoc test is frequently performed. The most common procedure is Tukey’s test. This helps researchers identify the location of the difference(s).