A shopper leaves a $105 cart behind. You send her a 15% code, and she buys. The report says the code recovered a sale. It cost $15.75.

Was it her code that did it? Some shoppers who abandon a cart come back an hour later on their own. For them, the code was a gift. The report can't tell the difference, because it only looks at the people who received the code.

What a holdout does

A holdout is a random slice of customers who get nothing. They are chosen the same way as everyone else and treated the same, except for the one thing you are testing. At the end you compare the two groups. The gap between them is what your play caused. Everything else would have happened anyway.

This is the oldest tool in measurement, and it is the one most AI projects skip, because the first results look great without it. Dashboards that count conversions among treated customers will show a win almost every time.

Why AI makes it more important

A model that scores customers and acts on the scores can look brilliant for a simple reason: it picks the customers who were going to do the thing anyway. A churn model flags accounts that were leaving and a recovery model contacts shoppers who were returning. Each shows a high success rate. Neither has proved it changed anything.

The more confident the model, the more it needs a control group, because confidence and causation are different things.

How to run one

Before anything is built, we settle five things.

• The unit. Decide what gets randomized: a shopper, an account, a store, a region. Pick the unit at which the action actually happens. Randomize accounts if one person handles the whole account.
• The slice. A common starting point is 5% to 10% held out, larger if the effect you expect is small. A permanent holdout can be tiny once the program is proven.
• The outcome. Choose one main measure before you start, such as revenue per customer or accounts retained. Money beats clicks.
• The window. Run long enough to cover the natural cycle: a full renewal, a full purchase cycle, a full week at least. Early results flatter new things.
• The stopping rule. Agree the result that would make you stop, and write it down. After launch there is pressure to read ambiguous results kindly.

How big it needs to be

Small effects need big samples. A common rule of thumb for an experiment with ordinary power puts the sample per group near 16 times the variance of the outcome divided by the square of the smallest effect you care about. You don't need to do the arithmetic by hand, but the lesson is useful. Halve the effect you want to detect and you need about four times the sample.

For a business with few customers, such as a B2B firm with a few hundred accounts, that means only large effects will show. That is acceptable. A program that can't show a large effect hasn't earned the right to scale.

What goes wrong

A few mistakes come up often.

• Leaky groups. Held-out customers get contacted by another team, or exposed through a channel you forgot. Their results look like treatment.
• Lopsided groups. The split isn't random, and the treated group has more big customers. Check that both groups look alike before the test begins and that the split matches what you intended.
• Short windows. A lift measured over three days may fade by day thirty, and a novelty effect looks real until it isn't.
• Moving goals. The team changes the target after the first look. Write the outcome down first and keep it.
• Peeking. Checking daily and stopping when the number looks good produces false wins.

Keep one running

The holdout stays on after launch. Customers change, catalogs change and models drift, so the gap you measured in week four may not be the gap in month nine. A small permanent holdout keeps the number honest and gives finance a figure it can trust.

Where it pays off most

It pays off most where the action costs something: discounts, free shipping, outreach from a person, a human reviewing a return. Those are the places where a model that is merely busy can quietly lose money.

Our cart recovery field guide shows how a model separates shoppers a nudge would change from those it wouldn't, and stop discounting people who would buy anyway goes further on the coupon case. The churn white paper applies the same rule to B2B accounts.