An A/B test comparing two variants of a version.
Control Versus Variant
Every A/B test compares two variants, but how they relate depends on how you started the test:- Control versus variant. When you test against a version that is already live, the live setup becomes the control (the baseline, shown as Variant A). You edit the alternative, Variant B, to create the change you want to test. This is the most common A/B test.
- Variant versus variant. When you split a section on a draft that has no live baseline, the two sides are equal variants (Variant A and Variant B) with no control. Use this when you are building something new and want to compare two fresh ideas.
How Traffic Is Split
Traffic is always split 50/50 between the two variants while the test runs. The split is fixed and cannot be changed. When the test concludes, all traffic moves to the winning or chosen variant.How the Winner Is Decided
Mention Me uses a Bayesian statistics model rather than traditional statistical significance. Instead of waiting for a fixed sample size, it continuously assesses the probability to be best for each variant: how likely that variant is to be the better performer. Two things shape how the test runs, set when you create it:- Test metric. The measure that decides the winner, either Conversion rate or Share rate. Share rate suits lower-volume audiences; conversion rate suits higher volumes.
- Confidence target. How sure Mention Me must be before declaring a winner. You influence this through the test speed (Normal, Fast, or Fastest), where a higher confidence target takes longer but gives a more certain result. For tests with no live baseline, the confidence is set from your weekly order volume instead.
How a Test Concludes
A test ends when a winner is found or when it reaches its maximum duration. You decide in advance what happens in each case, in the When it concludes step:- A winner is found. The winning variant is promoted automatically.
- A better variant is found, but no clear winner. You choose: allocate to the best performer (recommended), keep the control, or pick a specific variant.
- There is no meaningful difference. You choose: keep the control, or allocate to the variant.
What to Know Before You Start
- Duration. For a test against a live baseline, Mention Me estimates the duration from your order volumes (a power analysis). A variant-versus-variant test with no baseline runs for a fixed 8 weeks.
- Editing. You cannot edit a version while its A/B test is live. Adjust the setup before you launch.
- Promotions. A time-limited promotion cannot run an A/B test.
See Also
A/B testing best practices
What to test, and proven ideas with real results.
Set up an A/B test
Set up an A/B test.
Manage an A/B test
Manage an A/B test.
Monitor an A/B test
Monitor an A/B test.