What does a 95% confidence level indicate?

• A. It guarantees the results are correct
• B. If the same A/B test were run 100 times, similar results would be expected 95 times out of 100
• C. It means 95 percent of the data is within the average
• D. It indicates a 95 percent probability that the null hypothesis is true

Answer: (B)

A 95% confidence level describes how reliable an interval estimate is across repeated experiments. If you could redo the A/B test many times and compute a confidence interval from each result, about 95% of those intervals would contain the true difference in performance between the two variants. It’s about the method’s long-run performance, not a guarantee for this single run.

This doesn’t mean the results are guaranteed correct right now, nor that 95% of your data lie inside the interval, and it isn’t about the probability that the null hypothesis is true. The interval width depends on sample size and variability: more data or less variability gives a narrower, more precise interval.