Part four. The Decay
Drift
Nothing changed and it stopped working. The world moved and the rule did not.
Read the transcript
An automation that worked on the day it was installed will stop working, not because it changed but because the world it encodes did. Input drift: the data arriving changes shape. A new source, a new format, a field that used to be populated. Population drift: the cases arriving change composition, and a rule tuned on last year meets this year and the exceptions are no longer exceptional. Definition drift: the organisation quietly changes what a word means, and nothing tells the system. None of it triggers an error. So detection has to be scheduled rather than triggered. A recurring audit on a date, against a refreshed gold standard, with a named owner. Treat a missed review as an incident. An unreviewed automation is an assertion about the world that nobody has checked.
An automation that worked on the day it was installed will stop working, not because it changed but because the world it encodes did.
Input drift: the data arriving changes shape. A new source, a new format, a field that used to be populated and now sometimes is not.
Population drift: the cases arriving change composition. A rule tuned on last year's customer mix meets this year's and the exceptions are no longer exceptional.
Definition drift: the organisation quietly changes what a word means. The definition of a qualified lead changes in a sales meeting and nothing tells the system.
The reason drift is dangerous rather than merely inconvenient is that none of it triggers an error. The system continues, correctly executing a rule that no longer matches the world, and the fourth irony guarantees that a reliable system is monitored least attentively.
Detection has to be scheduled rather than triggered. A recurring audit on a date, against a refreshed gold standard, with a named owner. Quarterly for most things. Monthly where the population moves fast.
Put the review date in the same place as the automation, and treat a missed review as an incident. An unreviewed automation is an assertion about the world that nobody has checked.
Two things worth doing, neither of which costs anything
Read the standard in full, free and in the open, before you pay anything or give anyone an address.