Part three. The OwnerLesson 12 of 15
The manual test
Compute one important number by hand. What you learn is rarely the number.
The instrument The Manual Test
Read the transcript
Take one number the business runs on and produce it yourself, from source data, without using the report that normally produces it. The number will usually be close, and that is not the finding. The finding is everything you had to decide along the way. You will hit definitional forks the report resolved silently. Does revenue include the shipping the customer paid. Are refunds netted in the month of sale or the month of refund. Whichever way the report resolved these, it resolved them, and whoever built it made a judgment that is now organizational fact and is written down nowhere. Do this once a year for the two or three numbers that most drive decisions. And when the manual number disagrees, resist the urge to find your own error first.
Take one number the business runs on and produce it yourself, from source data, without using the report that normally produces it.
The number will usually be close, and that is not the finding. The finding is everything you had to decide along the way.
You will hit definitional forks the report resolved silently. Does revenue include the shipping the customer paid. Are refunds netted in the month of sale or the month of refund. Does a customer who bought in two segments count once or twice. Whichever way the report resolved these, it resolved them, and whoever built it made a judgment that is now organizational fact and is written down nowhere.
You will also find the joins. Numbers that cross systems are assembled somewhere, and the assembly usually contains at least one assumption that was true when it was built.
Do this once a year for the two or three numbers that most drive decisions. It is the only reliable way to find out what your reporting actually means, as opposed to what its column headings say, and it takes a morning.
The related discipline: when the manual number and the reported number disagree, resist the urge to find your own error first. Sometimes it is yours. Often enough it is not, and the instinct to assume the system is right is exactly why nobody checks.
Two things worth doing, neither of which costs anything
Read the standard in full, free and in the open, before you pay anything or give anyone an address.