AI Strategy 2 - Observability

Nothing can be improved unless it can be measured and nothing can be measured unless it can be observed. Most businesses have large parts that have never been observed, at least not documented, and therefore, never measured. They have accounts, a customer relationship management system (CRM), an organisation chart and a folder of written procedures, and they take that collection for a description of how the work is done. But, it isn’t. It is a description of how the work was intended to be done, assembled at different times by different people for different purposes, and rarely gets verified against how things actually get done.

Observability, for our purposes, means something narrower than it sounds. Everything that happens in a business is visible to someone. Somebody knows how the invoice gets raised, which spreadsheet the forecast actually lives in, who has to be asked before a price is agreed, and what happens when the usual person is on leave. It is all visible in the sense that it is visible to one person at a time, in fragments, and mostly in their heads. Observability is the condition in which that knowledge is recorded instead - written down, in one place, in such a way that someone who was not there can fully understand.

That gives us two separate tests - whether a thing can be seen at all by anybody and whether it has been seen and written down. Almost everything in a business passes the first test and a great deal of it fails the second. The audit we perform as the first engagement with a new client exists to close that gap. The size of the gap it finds is itself one of the more useful things to report.

Before the audit can start, the unit has to be chosen - the whole business if it is small enough, a division, a single role, or even a single outcome, if it is sufficiently complex. The method does not change with the size of the unit, which is the point we made at the outset and which the rest of this chapter demonstrates. The only thing that changes is the boundary, i.e. a handover from a person or application inside the unit to one outside. Beyond the boundary we record that the work leaves and where it goes to, and that’s all.

We audit large businesses one division at a time, often one role at a time, and assemble the parts afterwards. There is no other way to do it that produces anything usable, and attempting a single pass over everything at once is how a business ends up with the complete, accurate and unusable document described previously.

Our audit process has three parts:

1. map the people, and what each of them owns;
2. map the processes, including the systems they run in and the written procedure where one exists; and
3. map the handovers between them, step by step, across every flow.

The handovers are where we discover most of what matters. A business generally understands its people and has some idea of its systems. What it does not have is an account of what passes between them - who sends what to whom, in what form, by what route, and what the receiver has to do to it before it is usable. That is where the same data gets typed in a second time, where the version that everybody works from stops being the version of record, and where issues arise. The opposite condition is information captured once, in the flow of the work, in a form the rest of the business can use without entering it a second time.

The data map page draws these handovers between the steps, on a worked example built from an invented business.

We note three pieces of important information for each step - where it departs from the business’s own written process (workarounds), where the step re-enters data that already exists somewhere else (redundancy), and where the statement in the map is an inference rather than something observed. That third point is the one that is often overlooked. An audit that does not distinguish what was actually observed from what it deduced or accepted as reported is unreliable.

Some parts of a business are habitually unobserved, and they are the same parts in every business we have observed.

The first is judgement that runs in no system. In one of our audits, the step that decides whether a change arriving mid-cycle is material enough to act on is the only step in the entire map that touches no application at all. It happens in one person’s head, against a threshold that person set themselves, and the threshold is not derived from anything the business has measured. Where the item is written down when the decision is to defer it had never been established. A step like that is invisible to any survey of systems, because it is not in a system! It is only discovered by asking a person what they do and then asking what they do next and why.

The second is work that is intention rather than practice. Ask what the process is and you are told the process as it is supposed to be, i.e. what is written in the Standard Operating Procedures. However, ask what actually happened on the most recent occasions and you get something else. Both answers are given in good faith. The gap between them is not dishonesty, it is the ordinary drift of work under pressure, and it is entirely invisible to anybody reading the SOP.

The third is knowledge held in someone’s head. In one audit, the logic for composing a reference code, which a great deal of other work depends on, was not written down anywhere and was actually devised by someone who had since left the business! The consequence was not merely that the business was exposed once that person left, but that a code created wrongly could not be detected as wrong by anybody else. As it happens, in that role there was no written process for the work at all - no procedure note, no prescribed application, no prescribed store, and nothing a new joiner or a cover could read.

The fourth is anything that has never failed. A reconciliation done by eye, a settlement confirmed by checking the bank from time to time and holding the comparison in the head, a spreadsheet that one person maintains and everybody trusts. These are not mapped because nobody has ever had to ask about them. They surface in an audit and in an incident. Better that they surface in the audit!

The fifth is anything nobody owns. Access is the standard case. Who is able to export the business’s most valuable body of records had, in one engagement, never been established. In another, who holds read and write access to the records the unit works from was recorded as not established, and whether daily use of an assistant on company data sat inside a business agreement or on a personal account was recorded as not established either.

The reason all five stay unobserved is the same. Nobody is ever asked. The work is being done, the results appear, and no part of the ordinary running of the business requires anybody to describe how. Observation is a deliberate act, and it does not happen as a by-product of the work going well. Observation is useful in and of itself. As a foundation for the effective implementation of AI, it is essential.

Once the parts are mapped they are assembled. Two units mapped separately will sometimes produce maps that do not reconcile - the same activity described different ways by different people, the same record named two different things, the same thing described or recorded in two different ways, a handover that one side records and the other does not, a step one map treats as automatic and the other treats as manual.

The temptation is to tidy this away before anybody sees it. But, a mismatch between two honest maps is a useful insight, and usually a better one than anything either map contains on its own. It means the two parts of the business hold different accounts of the same work. As with the objectives in the previous chapter, it may be a legitimate difference of level or it may be a genuine defect, and the two are only distinguishable once both versions are recorded.

The audit produces data and information that are dealt with in the following chapters. Chapter three takes the data - every fact the unit holds, where it originates, its format, where and how it is stored, and who can access it. Chapter four deals with the processing (information synthesis) - how that data is cleaned, organised and given context, where that happens, and whether it is a defined process or a habit. Knowledge and judgement follow from those, and each of them needs the underlying layer to have been observed properly first.

Skip the audit and every chapter after this one is built on a description of the business that nobody has verified or validated. Do the audit without bounds and it produces the document from the previous chapter, complete and useless. Whatever the unit, the findings come back as four kinds: information duplicated, fragmented, missing, or inconsistently defined. Do it against the objectives, mark what was inferred, list what was not examined, and the material the rest of the series requires is in place.