Hypothesis Testing

How the model works

Each simulated experiment compares a treatment group with a control group (n per group, values with standard deviation 1). A two-sample z-test gives a p-value: the probability of a difference at least this large if there were no real effect (the null hypothesis).

• Declaring “significant” when p < α means that, under the null, a fraction α of experiments are false positives (Type I errors).
• Power is the chance of detecting a real effect of size d (in SD units). Approximately power ≈ Φ(d√(n/2) − z₁₋α/₂). It grows with effect size and sample size.
• Under the null, p-values are uniformly distributed between 0 and 1. With a real effect, they pile up near 0.

The “peek and stop” option checks after every 10 participants and stops as soon as p < α. That inflates false positives well above α, a common real-world error.

Normal data with known variance (z-test) for clarity. Real studies often use t-tests and face many other issues.

Guided experiments

1. No effect, α = 0.05. Run 2000 experiments with d = 0. Prediction: about 5% come out “significant”, all false positives. The p-value histogram is flat. Null true
2. Underpowered study. d = 0.3, n = 20 per group. Prediction: power is only about 16%; most real effects are missed. Small study
3. Peeking. d = 0 with “peek and stop”. Prediction: false positives climb to around 20–25%, far above the nominal 5%. Peek and stop

Challenge

How many participants per group do you need for 80% power to detect d = 0.5 at α = 0.05 (two-sided)?

References

• OpenStax, Introductory Statistics, ch. 9 “Hypothesis Testing with One Sample”.
• Simmons, Nelson & Simonsohn (2011), “False-positive psychology”, Psychological Science 22: 1359–1366.