Part one. The Reading
The holdout
To know whether your intervention works, some accounts must not receive it.
Read the transcript
Customer success programmes are almost never evaluated against a counterfactual, which means most claims about their effect cannot be falsified. The usual evidence is that accounts receiving the programme retained better than accounts that did not. That comparison is contaminated in the obvious direction: programmes are aimed at the larger or more engaged accounts, and those would have retained better anyway. So take the population you would enrol, randomise, and withhold from a slice. The objection is immediate and sincere, that it feels like deliberately harming customers. You are withholding something whose effect is unknown, which is the entire reason for the exercise. Keep it small, keep it time bound, and write down beforehand what you expect and how long the window is.
Customer success programmes are almost never evaluated against a counterfactual, which means most claims about their effect are unfalsifiable.
The standard evidence is that accounts receiving the programme retained better than accounts that did not. That comparison is contaminated in the obvious direction: programmes are aimed at accounts that were selected somehow, usually the larger or more engaged ones, and those accounts would have retained better anyway.
A holdout fixes it. Take the population you would enrol, randomise, and withhold from a slice. Then compare.
The objection is immediate and sincere: it feels like deliberately harming customers. It is worth answering honestly. You are withholding an intervention whose effect is unknown, which is the entire reason for the exercise. If it works you will know, and you will be able to fund it properly. If it does not, you have been spending people's time on it for years.
Keep it small, keep it time bound, and pre-register what you expect and how long the window is. A holdout examined whenever someone feels like it will be stopped the first week the numbers look bad, which is noise.
This is the same instrument as the marketing holdout in Volume III, and for the same reason: proximity is not causation in either function.
Two things worth doing, neither of which costs anything
Read the standard in full, free and in the open, before you pay anything or give anyone an address.