Estimate Statistical Power and Required Sample Size

Power is the probability of catching a real effect when one exists. Formally, 1 − β, the complement of the Type II error rate. A study with 60% power against the effect size you actually care about will miss it 40% of the time even when the effect is real. Underpowered studies are the main reason the literature fills up with false negatives that look like "no effect."

Four quantities are linked: sample size n, effect size (Cohen's d for means, h for proportions), significance level α, and power. Fix any three and the fourth is determined. Two modes you'll actually use: solve for n given a target power (0.80 is conventional, 0.90 if you can afford it), or solve for power given a fixed n and effect size. The math runs through the noncentral t for t-tests and noncentral F for ANOVA. The recurring trap worth flagging: don't estimate effect size from your own pilot data and feed it back into the power calculation. Pilot estimates are noisy and biased upward by selection. Anchor on a domain-driven "smallest effect of interest" instead.

A power calculator for hypothesis tests asks you to specify the effect size you want to detect before you run the study. Effect size is the magnitude of the difference between null and alternative hypotheses—maybe a 5-point IQ gain, a 2-millimeter reduction in tumor diameter, or a 0.3-unit shift in customer satisfaction score. Choosing this number forces you to think about what matters practically, not just statistically.

Standardized effect sizes like Cohen's d express the difference in units of standard deviation. A d of 0.2 is considered small, 0.5 medium, and 0.8 large. These benchmarks come from behavioral science, where Cohen cataloged typical study outcomes. But domain matters: a "small" effect in psychology might be clinically meaningful in medicine or trivial in physics. Use field-specific baselines when available.

Picking an unrealistically large effect size will give you optimistic power estimates and undersized samples. The study then fails to detect a real but smaller effect, wasting resources. Picking too small an effect inflates sample requirements beyond budget. The sweet spot is the minimum effect that would actually change a decision—what some call the "smallest effect of interest."

Suppose a new drug costs twice as much as the current standard. A 5% improvement might not justify the price; only a 15% improvement would. That 15% becomes your target effect size. Power analysis then tells you how many patients you need to detect it reliably. The calculation ties statistical planning to real-world stakes.

Alpha is the probability of rejecting the null hypothesis when it's actually true—a false positive or Type I error. Setting alpha to 0.05 means you accept a 5% chance of crying wolf. Regulatory agencies sometimes require alpha at 0.01 or even 0.005 for confirmatory trials, trading off power for stricter control of false claims.

One-tailed tests concentrate all rejection probability in a single direction. If theory firmly predicts a new therapy can only help—not harm—a one-tailed test at alpha 0.05 puts the entire 5% in the upper tail. This boosts power for detecting improvements but blinds you to worsening effects. Two-tailed tests split alpha evenly, detecting differences in either direction at the cost of slightly lower power per direction.

Most journal guidelines recommend two-tailed tests unless you can justify one-tailed a priori. "I expected the new method to be better" after seeing the data doesn't count—that's p-hacking. Pre-registration of one-tailed hypotheses protects against post-hoc rationalization.

Alpha, tails, and effect size interact in the power formula. Lowering alpha from 0.05 to 0.01 shrinks the rejection region, requiring larger samples to maintain the same power. Switching from two-tailed to one-tailed at fixed alpha increases power but limits generalizability. Understand these tradeoffs before committing to a design.

A power curve plots power—probability of detecting a true effect—against sample size or effect size. At small n, power hovers near alpha because you'd barely do better than chance. As n grows, the curve rises steeply, then flattens as power approaches 1.0. Most studies aim for power around 0.80, meaning an 80% chance of finding a real effect of the specified size.

The shape reveals diminishing returns. Jumping from n = 50 to n = 100 might lift power from 0.55 to 0.80—a big gain. But going from n = 200 to n = 400 might only bump power from 0.95 to 0.99. After a point, adding more data yields marginal benefit while doubling costs. The curve helps identify the "elbow" where investment efficiency peaks.

You can also fix n and vary effect size. Larger effects are easier to detect, so power climbs rapidly for big differences. The curve then shows the range of effects your study can reliably capture. If your minimum detectable effect exceeds what's practically meaningful, redesign before wasting resources.

Visualizing power curves clarifies tradeoffs better than a single number. Grant reviewers appreciate seeing how the proposed sample handles a range of plausible effects. Presenting curves also guards against "just significant" thinking—0.80 power is conventional, but 0.90 or 0.95 may be warranted for high-stakes decisions.

Power calculators typically offer two modes. In "solve for n" mode, you supply effect size, alpha, and target power—say, 0.80—and the calculator returns the sample size needed. This is standard for grant proposals and protocol planning, where you need a concrete recruitment target.

In "solve for power" mode, you fix sample size—perhaps constrained by budget or available patients—and the calculator tells you the power you'll achieve. If the answer comes back at 0.45, you face a hard choice: accept high risk of missing a real effect, or cut costs elsewhere to boost enrollment.

Some researchers run both modes iteratively. They start by asking how many subjects they can realistically recruit, check power, then adjust effect size assumptions or alpha to see if acceptable power is attainable. This exploratory dance often reveals whether the study is even feasible under budget constraints.

Neither mode changes the underlying relationship: power depends on effect size, variability, alpha, and n. Solving for one fixes the others. Understanding this constraint prevents magical thinking—you can't conjure power from thin air without bigger effects or bigger samples.

Researchers often plug in optimistic effect sizes pulled from pilot studies or published literature. But pilot studies are noisy, and published effects suffer from winner's curse—only the largest estimates clear the publication bar. Using inflated inputs produces undersized samples that fail to replicate the expected result.

Another mistake is ignoring attrition. If 20% of enrolled participants drop out, your final sample shrinks accordingly. Power analysis should target the post-attrition n, not the enrollment figure. Ignoring dropouts can turn a well-powered design into an underpowered mess once data collection finishes.

Using the wrong variance estimate is equally dangerous. If prior studies measured a different population or used different instruments, their variance may not transfer. Underestimating variability inflates apparent power; overestimating it wastes resources on excess enrollment.

Finally, some planners pick one-tailed tests to juice power without theoretical justification. Reviewers catch this gambit, and it can undermine credibility. If you can't defend directionality before data collection, stick with two-tailed and budget for the extra sample accordingly.