Under what condition is an A/B test considered statistically meaningful?

• A. When the results are visually compelling
• B. When the sample size is large enough for statistical significance
• C. When the test runs for a long period
• D. When the cost is minimal

Answer: (B)

In A/B testing, results are considered statistically meaningful when you have enough data to reliably distinguish a real difference from random noise. That means the sample size must be large enough to achieve statistical significance, given the expected effect and variability. A power calculation is typically done beforehand to set the minimum detectable effect, desired significance level (alpha), and desired power (often around 80%). If the collected data reaches that required size, you can be confident that the observed difference isn’t just due to chance.

Visually compelling results can be deceptive because human perception can read patterns into randomness. Running the test longer helps reduce random fluctuation, but duration alone doesn’t guarantee significance unless enough data is gathered. Cost being low doesn’t affect the statistical validity of the result.