HRO 15c: HRO Principle 7: Assessment Pt1

Introduction

A fundamental presumption of High Reliability Organizing (HRO), borne by experience, is that system and process design in practice don’t always conform to assumptions. Humans always get the final say in the struggle for compliance framed by the ineluctable conflict between values, goals, and environmental demands of work. HRO uses quantitative indicators to establish a formal baseline against which to identify reliability drift-movement toward conditions of greater risk. Assessment focuses on identifying and understanding the gap between work as imagined and work as performed. Assessment can produce surprise, which is a signal for the existence of something deeper in the organization that needs investigation to maintain safety. Managing encounters with this kind of surprise is not addressed by Weick and Sutcliffe’s five principles for HRO (Weick and Sutcliffe, 2015).

This post begins with what assessment is and continues by describing the concept and inevitability of drift away from design baselines. I will have more to say in a second post on assessment. The references below apply to this post and the second. I split the ideas into two posts because the essay exceeded (by far) my nominal length guidelines.

Assessment

Assessment carefully reviews what is documented (records) and what is done (observation) in the course of working. Assessment is the “Inspection” step of Shewhart’s conception of statistical quality control (Shewhart, 1939) and its recasting as the “Check” of the Plan-Do-Check-Act cycle (Deming, 1986).

Organizational assessment falls along a spectrum, from quality assurance by external authorities to evaluations performed to improve practice (Rhydderch, Edwards, Elwyn, Marshall, Engels,Van den Hombergh, & Grol, 2005). External assessments are done by supervisory authorities to assess compliance with regulatory standards. Assessment to improve day-to-day operations is usually done by an organization for personnel and process development to identify and correct problems locally. This is typically called self-assessment. HRO depends on both external and self-assessments.

Assessment compares organizational plans with what members of organization produce. It is “a structured method of collecting and evaluating information” about operations (Immordino, 2009). For HRO, it isn’t merely about evaluating areas of performance theorized to support high performance (organizational excellence), although that is one aspect. Assessments go beyond what an organization should do to excel to include what higher authority requires it to do.

Assessment focuses on fundamentals. HRO assessments seek a deep understanding of the organization by probing the micro-processes that support reliability: technical knowledge and training, qualification of operators, material readiness (including cleanliness), regulatory compliance, problem response, and administrative processes. A fundamental premise of HRO is that superior performance is based on healthy processes and deep knowledge and not the absence of bad outcomes or merely meeting operational goals.

Assessment is regular. It might be based on calendar periodicity or precede important events in an organization’s schedule such as assessing the fuel system maintenance, management, and processes of a ship before it conducts flight operations. Conducting periodic assessments enables higher authorities to incorporate lessons learned and improvements from recent assessments at other organizations.

Assessment is formal. It is based on objective criteria sourced to technical requirements. Assessments are documented in writing. Written assessment findings give an organization the opportunity to reflect on its operations, strengths, and opportunities for improvement (Immordino, 2009). The organization being assessed may be required to respond in writing to the findings depending on how their perceived seriousness. Written assessments support organizational analysis and corrective action. Documented assessments can be applied to other organizations as well as supply the information needed to make changes.

Drift

Organizational assessment is necessary for HRO because no plan or design remains unchanged under the pressure of operations.

“No plan of operations extends with any certainty beyond the first encounter with the main enemy forces” (von Moltke, 1871).

Drift is the "slow uncoupling of local practice from written procedure" (Snook, 2002, p.225). In any organizational design, processes can be judged unworkable by the people executing them. What matters for reliability in practice is how people at the working level think and operate, not what designers intended. Verbatim compliance may be desired by higher authority, but people develop their own ideas of what is required to accomplish organizational goals with more salience like efficiency and cost reduction (Dekker, 2011) or accomplishing non-nuclear tests on a commercial nuclear power plant (Chernobyl). They do this even when it can involve bypassing risk controls and defenses (ICAO, 2018; Malinauskas,  1987). Misconceptions and errors can be passed from one worker to another in the course of on-the-job training. Management isn’t informed about some change or problem even though they could be fixed without deviating from procedures.

There will always be practical drift. In organizations with complex interrelations between subgroups, subgroup norms that might not conform to organizational norms, communication gaps, changing technical requirements, goal conflicts (there are always goal conflicts), and new people constantly in need of training (often done badly), there will always be different interpretations of priorities and rules (Angle & Perry, 1981).

“Contradictory goals are the rule, not the exception, in complex situations” (Dörner, 1989, p.65)

Judgments and decisions about what is acceptable are strongly influenced by production pressure, staffing levels, budgets, workloads, experience, and qualifications (Dekker, 2011).

Something that I haven’t seen addressed in academic literature is HRO practices aren’t easy to do consistently well. They require expertise and considerable strength of character. Technical training, interactive briefings, planning responses to failure, and operationalizing questioning attitude are difficult and can fall apart under toxic leadership styles or contexts where leaders exhibit low standards. Even assessments can be perfunctory, incurious, and have poor preparation of auditors and non-existent follow-up. This is why even they need to be assessed.

The crew and Commanding Officer of the USS JOHN S MCCAIN did many things associated with HRO before the collision with the tanker ALNIC MC in 2017: navigation briefing, Commanding Officer stationing himself on the bridge, and three officers questioning the CO’s decision to enter the Traffic Separation Scheme without setting the Sea and Anchor Detail, but they weren’t done well and were undermined by many other deficiencies in training, experience, formality, qualifications, and technical documentation (Pettersen & Schulman, 2019). It is not enough to do HRO practices, an organization has to do them well consistently. Since collisions at sea and other disasters are rare events, a lack of disastrous outcomes is no reason for confidence.  This is the essence of Preoccupation with Failure (Weick & Sutcliffe, 2015).

The inevitability of drift from baselines does not mean that technical requirements, operational plans, and procedures are always correct. They are not (NTSB, 2019). This is why many oversight and design organizations strongly encourage feedback from operators. The problem is that flexibility, feedback with speedy replies, technical rigor, and compliance are competing goals. Assessments are the only way to learn how well an organization is managing them.

In part two of my essay on assessment, I will address its contribution to resilience and the components of HRO assessments.

Assessment References (applicable to both posts)

Angle, H. L., & Perry, J. L. (1981). An empirical assessment of organizational commitment and organizational effectiveness. Administrative Science Quarterly, 26(1), 1-14.

Bierly III, P. E., & Spender, J. C. (1995). Culture and high reliability organizations: The case of the nuclear submarine. Journal of Management, 21(4), 639-656.

Dekker, S.W.A. (2011). Drift into failure: From hunting broken components to understanding complex systems. Ashgate Publishing Co.

Deming, W.E. 1986. Out of the crisis. MIT Press.

Dörner, D. 1989. The logic of failure: Recognizing and avoiding error in complex situations. Perseus Books.

Helmreich, R.L., Wilhelm, J.A., Klinect, J.R., & Merritt, A.C. (2000). Culture, error and crew resource management. In E. Salas, C.A. Bowers, & E. Edens (Eds.), Applying resource management in organizations: A guide for professionals. Erlbaum.

Immordino, K. M. (2009). Organizational assessment and improvement in the public sector. CRC Press.

International Civil Aviation Organization (ICAO) (2018). Safety management manual (4th ed). Doc 9859. Retrieved from https://skybrary.aero/sites/default/files/bookshelf/5863.pdf

Malinauskas, A. P. (1987). The Chernobyl accident: Causes and consequences (No. CONF-8709104-3). Oak Ridge National Lab. Retrieved from https://www.osti.gov/servlets/purl/6156211

Moltke, H. (1871). Moltke’s military works: II. Activity as chief of the army general staff in peacetime, second part. Ernst Siegfried Mittler und Sohn.

Pettersen, K. A., & Schulman, P. R. (2019). Drift, adaptation, resilience and reliability: Toward an empirical clarification. Safety Science, 117, 460-468.

Rhydderch, M., Edwards, A., Elwyn, G., Marshall, M., Engels, Y., Van den Hombergh, P., & Grol, R. (2005). Organizational assessment in general practice: a systematic review and implications for quality improvement. Journal of Evaluation in Clinical Practice, 11(4), 366-378.

Shewhart, W. A. (1939). Statistical method from the viewpoint of quality control. U.S. Department of Agriculture.

Snook, S. A. (2000). Friendly fire: The accidental shootdown of US black hawks over northern Iraq. Princeton university press.

Weick, K.E., Sutcliffe, K.M. (2015). Managing the unexpected: Assuring high performance in an age of complexity (3rd ed.). Jossey-Bass.

Wildavsky, A. (2017). Searching for safety. Transaction Books.