NBOP National Board of Operations Professionals

Part two. The StandardLesson 7 of 15

The gold standard problem

You cannot tell whether output is right without a definition of right that existed beforehand.

The instrument The gold standard

0:51. Captions on by default. Everything said here is also written on this page, so nothing is only in the video.

Read the transcript

To evaluate any automated output you need examples of correct output, produced independently, before you see what the system does. That set is the gold standard, and most operations do not have one. Without it, evaluation collapses into plausibility, and plausibility is exactly what a well built automated system is best at producing, including when it is wrong. Building one is unglamorous. Take fifty real cases. Have the most capable person you have produce the correct output for each, without reference to the system. Where they are unsure, record that too, because cases a human expert finds ambiguous are cases you should not be automating unattended. Then hold it back. A gold standard the people configuring it can see stops being a test and becomes a target.

To evaluate any automated output you need examples of correct output, produced independently, before you see what the system does. That set is the gold standard, and most operations do not have one.

Without it, evaluation collapses into plausibility. Somebody looks at the output, it seems reasonable, and it is accepted. Plausibility is exactly the thing a well built automated system is best at producing, including when it is wrong.

Building one is unglamorous. Take fifty real cases. Have the most capable person you have produce the correct output for each, without reference to the system. Where they are unsure, record that too, because cases a human expert finds ambiguous are cases you should not be automating unattended.

Then hold it back. A gold standard that the people configuring the system can see stops being a test and becomes a target, and you will get a system tuned to those fifty cases.

Refresh it periodically, because the world moves and a standard built three years ago encodes what was correct three years ago. That is the same failure this course is about, applied to the instrument that is supposed to detect it.

Two things worth doing, neither of which costs anything

One Help decide what a competent operator is expected to know The eight domains carry equal provisional weighting until practicing operators say what the work actually is. Takes about eight minutes. Two Check whether you are eligible About a minute. No account, no email field, and nothing you enter is kept.

Read the standard in full, free and in the open, before you pay anything or give anyone an address.