What is statistical significance?

• A. The determination of whether test results are due to random chance or if they reflect a true effect
• B. The exact p-value of the test
• C. The size of the sample required
• D. The proportion of users who convert in the control group

Answer: (A)

Statistical significance is the determination of whether observed test results are due to random chance or reflect a true effect. In practice you set a significance level (often 0.05) and compute a p-value. If the p-value is below that threshold, you conclude the result is statistically significant, meaning the evidence against the null hypothesis is strong enough to reject it at that level. This concept is about deciding whether an effect is likely real, not about how big the effect is. The exact p-value informs the strength of the evidence, but significance is the conclusion you reach from that p-value. Also, significance depends on sample size: very large samples can show significance for tiny differences that may not matter in practice, while small samples might not show significance even for meaningful effects. The other options describe the exact p-value, the required sample size, or a specific conversion rate, none of which define what significance means.