Part two. The Standard
The demonstration problem
A demonstration is chosen. An audit is not. Only one of them tells you anything.
Read the transcript
Every automation is demonstrated before it is adopted, and every demonstration uses examples somebody selected. Even with no intent to mislead, the selection leans toward cases that work, because those are the cases used during building. The distinction is worth being pedantic about. A demonstration answers whether the system can produce a correct result. An audit answers how often it does, on work you did not choose. So take a random sample of real cases from last month, before anybody knows which ones. Run them. Compare against the gold standard. Report the rate and, more importantly, the shape of the failures. Failures clustered on one customer type or one input format are a systematic gap, and a sampling check will miss them exactly when it matters.
Every automation is demonstrated before it is adopted, and every demonstration uses examples somebody selected. Even with no intent to mislead, the selection is toward cases that work, because those are the cases that were used during building.
The distinction is simple and worth being pedantic about. A demonstration answers whether the system can produce a correct result. An audit answers how often it does, on work you did not choose.
The audit procedure: take a random sample of real cases from the last month, before anybody knows which ones. Run them. Compare against the gold standard. Report the rate and, more importantly, the shape of the failures.
The shape matters more than the rate because it tells you whether the failures are random or clustered. Random failures at a low rate may be tolerable with a sampling check. Failures clustered on a particular customer type, or a particular input format, are a systematic gap, and the sampling check will miss them exactly when it matters.
A vendor demonstration is not an audit, and asking a vendor to run an audit on your data with your gold standard is a reasonable request that changes conversations quickly.
Two things worth doing, neither of which costs anything
Read the standard in full, free and in the open, before you pay anything or give anyone an address.