How do you define success for an A/B test?

• A. By the exact number of emails sent
• B. By agreeing on the expected improvement percentage, such as a 5% or 10% increase
• C. By the size of the audience
• D. By the speed of the test completion

Answer: (B)

Defining success for an A/B test means agreeing in advance on what amount of improvement in the chosen metric would count as a meaningful win. You pick the metric you care about (for example, conversion rate, click-through rate, or revenue per visitor) and set a target uplift, such as a 5% or 10% increase. This predefined uplift becomes your decision rule: if the observed difference between the variants reaches that improvement with enough statistical confidence, you implement the winner; if not, you revert or iterate.

The exact number of emails sent, the size of the audience, or how quickly the test finishes aren’t success criteria themselves—they’re inputs that affect the test’s ability to detect an uplift with reliability. The key is the agreed-upon improvement threshold and the accompanying confidence level to declare a result trustworthy. This upfront definition keeps the test objective and prevents post hoc or subjective decisions.