Bainbridge's ironies

Automation gives the human the tasks it could not do, and then expects them to catch what it gets wrong.

In nineteen eighty three Lisanne Bainbridge published a short paper on the ironies of automation, working from process control and aviation. It has aged extraordinarily well and it describes most of what goes wrong with software automation today.

The first irony: the designer who automates because humans are unreliable is themselves human, and their errors are now built into the system and reproduced perfectly at scale.

The second: automation takes the easy, routine parts and leaves the operator the hard, rare, ambiguous ones, which are the parts requiring most skill, now performed without the routine practice that maintained that skill.

The third: skill decays without use. An operator asked to take over in an emergency is asked to perform, under pressure, a task they have not practised since the system was installed.

The fourth: monitoring is something humans are demonstrably bad at. Watching a reliable process for a rare fault is a task attention is unsuited to, and the more reliable the system the worse the monitoring, because vigilance follows expectation.

The operational consequences are direct. Keep some routine work manual to maintain skill. Rehearse takeover before it is needed. Do not build a control that depends on a human noticing something rare. And design the failure signal to arrive rather than to be observed.

Two things worth doing, neither of which costs anything