Compare Three or More Group Means With ANOVA

Family-wise error inflation. Three groups means three pairwise comparisons, four groups means six. At α = 0.05 per test, the probability of at least one false positive across three comparisons is roughly 14%. ANOVA wraps the omnibus comparison into one F-test that holds the overall α at 0.05. Once F is significant, you run a post-hoc procedure that controls the family-wise error: Tukey HSD, Bonferroni, Scheffé, or Holm-Bonferroni. The post-hoc step is what tells you which groups differ; ANOVA only flags that something does.

Tukey HSD when sample sizes and variances are roughly equal. It's the standard choice and controls family-wise error tightly. Games-Howell when variances differ across groups; it's the heteroscedastic counterpart of Tukey. Bonferroni when you have a small number of pre-specified comparisons (it's conservative for many comparisons). Holm-Bonferroni is uniformly more powerful than plain Bonferroni and is a free upgrade. Dunnett when comparing several groups against one control. Don't run pairwise t-tests with no correction after ANOVA.

η² = SSB / SST, the fraction of total variance explained by group membership. Cohen's benchmarks (1988): 0.01 small, 0.06 medium, 0.14 large. η² is upward-biased at small samples; partial η² and ω² (omega-squared) are bias-corrected alternatives. ω² = (SSB − (k − 1)·MSE) / (SST + MSE) is closer to the population effect size and can be slightly negative when the effect is genuinely zero. Report alongside the F statistic and p-value, especially with small group sizes where significance can outpace meaningful effect.

Use Welch's ANOVA, which doesn't assume homogeneity of variance. R: oneway.test(y ~ group) defaults to Welch's. SPSS exposes it as a checkbox. Standard ANOVA tolerates moderate variance differences when group sizes are roughly equal, but with unequal variances and unequal n, the Type I error rate can be substantially off (either inflated or deflated depending on the direction). Levene's test or Brown-Forsythe screens for homogeneity, but defaulting to Welch's avoids the screening step entirely.

One-way assumes independent observations across groups: each subject contributes to exactly one group. Repeated measures applies when the same subjects are measured under all conditions, so observations are paired across treatments. Treating repeated-measures data as if it were one-way independent inflates the variance estimate and robs the test of power. R: aov(y ~ treatment + Error(subject/treatment)) for repeated measures, or lme4::lmer(y ~ treatment + (1 | subject)) for the mixed-effects equivalent. SPSS has a separate repeated-measures ANOVA module.

F = MSB / MSW, the ratio of between-group variance to within-group variance. Numerator df = k − 1 where k is the number of groups. Denominator df = N − k. F is reported as F(df_num, df_den) = X.XX. Under H₀ (all means equal), F follows the F distribution with those df, and the p-value is the right-tail. F < 1 means within-group variance exceeds between-group variance, which trivially gives p > 0.5; F much greater than 1 is what produces small p-values.

Standard format: "F(df_num, df_den) = X.XX, p = .XXX, η² = .XX." Example: "A one-way ANOVA showed a significant effect of treatment on yield, F(2, 87) = 7.42, p = .001, η² = .15. Tukey HSD post-hoc tests indicated that treatment B differed significantly from controls (p = .002, mean difference = 4.3, 95% CI [1.7, 6.9])." Include the omnibus F, the post-hoc results that drove the conclusion, and effect sizes (η² for the omnibus, mean differences with CIs for the pairwise comparisons).

ANOVA is fairly robust to non-normality at moderate group sizes (n ≥ 15-20 per group), thanks to the CLT. Severe skew or outliers at small n can distort the F-test. Kruskal-Wallis is the rank-based non-parametric alternative; Dunn's test is its post-hoc counterpart. Permutation ANOVA (R's coin::oneway_test) avoids both the normality and equal-variance assumptions. Don't quietly trim outliers; document any data exclusions, justify them, and ideally pre-register the rule.