Effect Size

In a study or experiment with two groups (usually control and treatment), the investigator typically has in mind the magnitude of the difference between the two groups that he or she wants to be able to detect in a hypothesis test. This magnitude, e.g. a 20% improvement in click-through rates, or a 20 point drop in blood pressure, is the effect size. Sample sizes are then chosen so that the probability of detecting this effect size, if it exists, is high. Too small a sample, and the effect may go unnoticed because it is not distinguishable from random variation. Too large a sample may guarantee detecting the effect, if it exists, but wastes resources.

See Sample Size Calculations.