Laws of Information 4 and 5

A long time ago, I proposed some ‘laws of information’, looking particularly at the kind of information available to manage large public organisation. These are as below:
1. Information is costly
2. Data is always less reliable than you think.
3. data that is collected to measure performance loses reliability
You can click these weblinks to access the relevant text for each law.

Here is another law:
4. There is always more information available than you first thought.
Natural science has advanced through ever-improved measurement techniques, of which the Large Hadron Collider is the most recent and by far the most expensive. Each area of science has its own preferred measurement technique, and great effort is expended in improving its accuracy and reliability. Social science works on very different principles: no single measure remotely approaches the levels of accuracy taken for granted in the natural sciences, and so social scientists draw on multiple sources of data to base their conclusions. This principle is also a good one for managing large public organisations like universities and hospitals, where there is also a mass of different information, sometimes of dubious reliability (for reasons why it is dubious, see Laws of Information 1-3).

Sadly, this principle is not always applied. Managers and politicians often focus on one (unreliable) measurement and ignore the others. In part, this is a consequence of a commitment to the written word. Nothing is truly believed to exist unless it has been written down, preferably on a form. Once written down, it is believed superior to all other forms of information such as observation, informal discussions with staff, or patients’ letters of complaint. The most holy of all written data is quantitative data, especially that emanating from a computer. This tendency is reinforced by the use by governments of simple quantitative targets to measure the complex activities of complex institutions.

As an example, look at the Stafford Hospital case. Analyses of routine data showed that the hospital was an outlier in mortality statistics for some surgical procedures (ie people were much more likely to die). This all explained in an excellent article in the London Review of Books, available here:
Rigging the death rate
If the hospital management had followed the Fourth Law of Information, they would have seen this analysis of mortality statistics as a sign that they should gather data from other sources. They could then have visited the wards and observed daily care, spoken to patients and staff, reviewed casenotes, checked how their staffing levels compared with those of other similar hospitals, or brought in some outside experts to do these things and advise. They don’t seem to have done any of this: instead, they decided to discredit the mortality statistics. Management consultants were brought in to change the diagnostic codes of patients who died in hospital. Researchers at Birmingham University were funded to discredit the use of statistics to assess hospital performance. Their report made the correct conclusion that statistics can be misleading and that one set of them should not be used exclusively to assess performance. But that of course misses the point. Being an outlier should be regarded as a warning sign rather than definitive proof. It should have indicated a need to collect other data. In other words, the truth is found not in one set of data, however tidily it is presented and however quantitative, but in a wide range of information, from which an informed person can make a judgement. This leads to the fifth law of information:
5. Interpreting information requires judgement.

The word ‘judgement’ of course will sound a warning bell to some. How much better to pretend that decisions follow automatically from the data without human intervention or the exercise of personal responsibility. Then all that is needed when things go wrong is for the relevant procedures to be blamed and amended. This defence (“I was only carrying out procedures”) is an effective life strategy in any large organisation, and may be a more reliable path to promotion than anticipating problems and taking the initiative in solving them. Look around, and you will see the consequences.