NBOP National Board of Operations Professionals

Before you start

It was working

Automation rarely fails loudly. It produces plausible output that nobody is positioned to question.

0:45. Captions on by default. Everything said here is also written above, so nothing is only in the video.

Read the transcript

When a person makes a mistake, the mistake usually looks like a mistake. It is inconsistent with the surrounding work, somebody notices, and it gets corrected. When an automated process fails, it usually keeps producing output in exactly the same format, at exactly the same rate, with exactly the same confidence. The output is wrong and nothing about its appearance says so. The classification rule that no longer matches reality still classifies everything. The report that lost a data source still produces a report. Because output continued, everyone downstream continued to rely on it, and by the time somebody notices, months of decisions rest on it. So the question is not whether something can be automated. It is: when this is wrong, who finds out, how, and how long does it take?

When a person makes a mistake, the mistake usually looks like a mistake. It is inconsistent with the surrounding work, somebody notices, and it gets corrected.

When an automated process fails, it usually keeps producing output in exactly the same format, at exactly the same rate, with exactly the same confidence. The output is wrong and nothing about its appearance says so.

That is the whole problem, and it is why automation failures are discovered late and in bulk. The classification rule that stopped matching reality still classifies everything. The report that lost a data source still produces a report. The model that drifted still returns an answer with the same shape as a correct one.

The organisational consequence is worse than the technical one. Because output continued, everyone downstream continued to rely on it, and by the time somebody notices, months of decisions rest on it.

So the question this volume keeps returning to is not whether something can be automated. It is: when this is wrong, who finds out, how, and how long does it take? An automation without an answer to that is not a solution, it is a deferred incident.

Two things worth doing, neither of which costs anything

One Help decide what a competent operator is expected to know The eight domains carry equal provisional weighting until practising operators say what the work actually is. Takes about eight minutes. Two Check whether you are eligible About a minute. No account, no email field, and nothing you enter is kept.

Read the standard in full, free and in the open, before you pay anything or give anyone an address.