Why a Holdout Rather Than an A/B Test
A holdout is not an A/B test arm, and A/B testing is not available on a product referral campaign. The Holdout node is the one node in the editor that is not A/B testable, for the same reason. An A/B test asks “which of these two offers is better?”; a holdout asks “is offering anything at all better than offering nothing?” Understanding Brand and Product Referral covers the distinction in full.Set the Holdout
The holdout is set in the Holdout node of the campaign editor, and again in the last step of the setup wizard when you create an incentivised campaign. The field is labelled Holdout (control) split, and its helper text reads “This share of visitors is held back from the incentive so you can measure the extra revenue it drives.” Enter a whole number between 1 and 99. The setup wizard defaults to 50%.The holdout belongs to the version, not the campaign. Each version carries its own, so the split always matches the version that is live. Creating a new version lets you change the split without touching the one customers are currently seeing.
Choosing a Split
There is a trade-off in both directions:- A larger holdout gives you a cleaner, faster read on the incentive’s effect, because the control group reaches a usable size sooner. It also means fewer people see the reward.
- A smaller holdout shows the reward to more people, but the comparison takes longer to become trustworthy, and on low-volume traffic it may never get there.
The Split Is Locked Once Live
Like every other node, the Holdout node is only editable on a draft version. Once the version is live it becomes read-only. This is deliberate. Changing the split part-way through invalidates the comparison, because the two groups would no longer have been treated consistently over the measurement period. To change it, create a new draft version, set the new split, and launch it.Non-Incentivised Versions Are a 100% Holdout
A non-incentivised version is, in effect, a holdout of everybody: no visitor is ever shown an offer. It has no Holdout node, because the field only accepts 1% to 99% and so could never represent it. See campaign modes.What You Can Measure Today
When it does arrive, the comparison will report on the full cohorts, everyone who was eligible, not just the people who took the offer up. Comparing offer-takers against the control group would flatter the result, because the people most likely to redeem an offer are also the people most likely to have bought anyway.See Also
Campaign modes
Non-incentivised and incentivised.
Set up a campaign
Where the holdout is set during setup.
Brand and product referral
Where holdouts sit among the referral concepts.
Launch a version
Change a split by launching a new version.