Document Sample
249 Powered By Docstoc
					Klinect, J.R., Wilhelm, J.A., & Helmreich, R.L. (1999). Threat and error management: Data from line operations safety audits.
  In Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 683-688). Columbus, OH: The Ohio State University.


                                James R. Klinect, John A. Wilhelm, and Robert L. Helmreich
                                      The University of Texas Team Research Project
                                                 Department of Psychology
                                                     Austin, Texas USA

                        ABSTRACT                                         Wilhelm, Klinect, & Jones, 1999). The observers
                                                                         were asked to (1) document external threats, (2) record
Using data collected from Line Operations Safety                         flightcrew errors in terms of their type, response, and
Audits (LOSA), this paper presents a quantitative view                   outcome, and (3) rate the crew on several CRM
of external threat and flightcrew error in normal flight                 behavioral markers. In sum, these three types of data
operations. Results on the different types of threat and                 have provide a unique insight for threat and error
error and their management responses will be                             management during normal operations.
presented. The results also indicate a substantial
amount of variability across error outcomes, phases of                                    EXTERNAL THREATS
flight, and on a larger scale, airlines. Implications and
future avenues of research with this type of proactive                      In the model of threat and error management,
data are also discussed.                                                 external threats are defined as situations, events, or
                                                                         errors that originate outside of the cockpit (Helmreich,
                    INTRODUCTION                                         Klinect, and Wilhelm, this volume).            Examples
                                                                         include high terrain, poor weather, aircraft system
   In the paper titled Models of Threat, Error, and                      malfunctions, or errors made by other humans in the
Response in Fight Operations, two models of threat                       system such as maintenance or ATC. By categorizing
and error management were outlined and discussed                         and counting such threats, it allows researchers and
(Helmreich, Klinect, Wilhelm, this volume). The                          airlines to quantify some of the risk that is associated
purpose of this paper is to present data that were                       with the flying environment.
collected under the framework of these models in
recent Line Operations Safety Audits (LOSA). The                            In the discussion of external threats, three questions
paper will be divided into five sections: LOSA                           are addressed to help organize the interpretation of the
demographics, external threats, threat management,                       results. The first question inquires about the extent of
flightcrew errors, and error management.                                 external threats in normal flight operations. The next
                                                                         question focuses on the phases of flight that are most
               LOSA DEMOGRAPHICS                                         prone to threats. Finally, the last question asks
                                                                         whether airlines have the same distribution or
   Between 1997 and 1998, the University of Texas                        exposure to external threats.
Team Research Project conducted LOSA’s at three
airlines.   The aggregated sample contains 184                           What is the extent of external threats in normal
flightcrews on 314 segments. Table 1 provides a                          flight operations?
demographic summary for each airline in the database.
                                                                            Judging from the aggregated LOSA results, external
      Table 1 – LOSA Demographics by Airline                             threats are common in normal flight operations. While
           91 flight segments observed                                   this probably no surprise, the LOSA data in Table 2
 Airline                                                                 roughly quantify the degree of threat that crews have
           Median stage length: 3 hrs. 15 min.
   A                                                                     to face on a daily basis.
           Stage length range: 45 min. to 14 hrs.
           102 flight segments observed
 Airline                                                                        Table 2- General External Threat Results
           Median stage length: 3 hrs. 30 min.
   B                                                                      Total external threats                     606
           Stage length: 45 min. to 12 hrs.
           121 flight segments observed                                   Percentage of flight segments with
 Airline                                                                                                                      72%
           Median stage length: 1 hr.                                     at least one external threat
           Stage length: 20 min. to 2 hrs. 35 min.
                                                                          Average external threats per flight
   During a LOSA, trained observers are asked to                          segment
record three types of data using the most recent                          Most external threats on one flight                  11
version of the Line/LOS Checklist (Helmreich,

Klinect, J.R., Wilhelm, J.A., & Helmreich, R.L. (1999). Threat and error management: Data from line operations safety audits.
  In Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 683-688). Columbus, OH: The Ohio State University.

   Data also show that the number of external threats                    different set of threat management strategies in certain
can vary from flight to flight. As seen in the table                     phases of flight.
above, the number of external threats per flight can be
extensive at times. In a flight that we label as the                     Are all airlines exposed to the same distribution of
“heavy rain of threat,” a flightcrew had to manage                       external threats?
eleven different threats.      Some of them were:
inconsistent fuel slips, weight restriction on takeoff,                    For the number of threats per flight segment, the
smoke in the lavatory, heavy traffic, adverse weather                    results indicate that the three airlines in the database
on departure and arrival, and a late runway change.                      were not exposed to the same distribution of external
This was no doubt a heavy workload for the crew and                      threats (Table 4). At a more specific level, the results
one that posed a significant risk to safety. On their                    also show significant airline differences in the types of
next leg, the crew got a partial break when they were                    external threats1. These included terrain, adverse
confronted with three external threats.                                  weather, aircraft malfunction, ground operation events,
                                                                         and external errors
   Future research into external threats will attempt to
build an objective metric for operational complexity                            Table 4 – External Threat Variability by Airline
on the basis of type and frequency. By using this                                                    Airline Airline       Airline
metric, our first step will be to validate previous                                                     A          B          C
research that has shown flightcrew performance tends                         Average number of
bifurcate from their baseline performance when                               external threats per      3.3        2.5         .4
operating in complex environments (Hines, 1998).                             flight segment

In what phase of flight do most external threats                            The preceding data is useful for airlines because it
occur?                                                                   allows them to target areas of external risk in an
                                                                         systematic fashion. In other words, it allows the
   While the number of external threats can drastically                  operational complexity for an entire airline to be
vary from flight to flight, they do exhibit some                         profiled. For example, Airline A operated in an
consistency in the sense that some phases of flight                      environment with proportionally more aircraft system
were more prone to have threats (Table 3).         The                   malfunctions, terrain threats, and adverse weather
highest percentage of external threats were in the                       threats.    In contrast, Airline B operated in an
descent / approach / landing phase of flight. These                      environment that had significantly more external
tended to be terrain, adverse weather, and ATC                           errors, ATC threats, and ground events than Airlines A
command threats such as abnormal routing, poor                           and C. Airline C, which flew into a relatively benign
timing, or slam-dunk clearances.                                         environment, had the fewest overall external threats
                                                                         per flight segment. These data could also be used to
    Table 3 – External Threats by Phase of Flight                        profile threats for particular geographic locations,
                                   Percentage of                         airports, or fleets.
        Phase of Flight
                                 External Threats
 Preflight / Taxi                      22%                                                 THREAT MANAGEMENT
 Takeoff / Climb                       28%
 Cruise                                10%                                  Threat management is the act of minimizing the
                                                                         potential consequences of threats on flight safety. To
 Descent / Approach / Land             39%
                                                                         explore the data on threat management in normal flight
 Taxi / Park                            1%                               operations, two questions will be addressed. The first
                                                                         concerns external threats with which flightcrews have
   It was surprising to note that 22% of the external                    the most difficulty in managing. The second question
threats occurred before the aircraft left the ground in                  covers the type of crew behaviors that were observed
the preflight / taxi phase of flight. These threats were                 to be the most important in threat management.
mostly associated with aircraft system abnormals,
ground operation events, and operational pressures                       Which external threats do flightcrews have the
such as delays, late aircraft arrival, or other irregular                most difficulty managing?
operations. The nature of the typical preflight / taxi
threats were substantially different from those that                        One of the potential consequences of an external
occur when the aircraft is airborne. Therefore, future                   threat can be the propagation of flightcrew error.
research will investigate whether flightcrews use a
                                                                             Chi squares were significant at the .05 level

Klinect, J.R., Wilhelm, J.A., & Helmreich, R.L. (1999). Threat and error management: Data from line operations safety audits.
  In Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 683-688). Columbus, OH: The Ohio State University.

While the linkage between threats and errors are not                             Table 5 - General Flightcrew Error Results
perfectly causal in nature, we do believe that we can                        Total errors                                             578
determine if an error would have occurred without the
presence of a threat. Based upon this assumption, we                         Percentage of flightcrews with at least
asked the observers to link external threats and                             one error
associated errors. The results show that only 7% of                          Percentage of flight segments with at
the threats in the database influenced flightcrews to                                                                                64%
                                                                             least one error
commit an error. Among this small number of
linkages, an interesting finding did come about. More                        Average errors per flight segment                       1.84
than half of the linkages were associations between
unusual ATC command threats, such as late runway                             Most errors on one flight segment                         14
changes or slam-dunk clearances, to flightcrew
decision errors that resulted in an unnecessary                             These data show that flightcrew errors were just as
acceptance of risky ATC instructions.                                    pervasive as external threats. Over 60% of the
                                                                         flightcrews and segments in the database contained
What flightcrew behaviors were observed to be the                        errors. There was also a substantial degree of
most important in threat management?                                     variability with flightcrew errors.

   For the past three line audits, we have asked our                     In what phase of flight do most errors occur?
observers to list flightcrew behaviors that either led to
effective or ineffective threat management. The                             The phases of flight most prone to external threats
results show that the most frequently mentioned                          were also most likely to contain flightcrew errors
behaviors were strong leadership through the                             (Table 6). As was the case with the external threats,
coordination of flight deck activities, active                           most of the flightcrew errors were committed in the
preparation for contingency situations, flightcrew                       descent / approach / landing phase of flight. Typical
vigilance, quality and content of briefings, and crew                    errors in this phase were associated with checklist
members that asked questions or spoke up with                            usage, automation, and proficiency. Interestingly this
pertinent information.                                                   phase also contained the most errors linked to
                                                                         consequential outcomes2. This is further evidence that
               FLIGHTCREW ERRORS                                         this phase of flight is the riskiest for flightcrews to
   Flightcrew errors are defined as a crew action or
inaction that leads to a deviation from crew or                              Table 6 – Distribution of Flightcrew Errors by Phase
organizaitonal intentions or expectations (Helmreich,                                              of Flight
Klinect, Wilhelm, this volume). Again, using the                                                                     Percentage
models of threat and error management as conceptual                                                  Percentage
                                                                                Phase of flight                       that were
foundations, the LOSA data allows us to examine                                                        of errors
flightcrew error in normal flight operations.                                Preflight / Taxi            23%             7%
                                                                             Takeoff / Climb             24%            12%
   In this section, four questions will lead the
discussion of the flightcrew error results. The first                        Cruise                      12%            12%
question asks about the extent of flightcrew errors.                         Descent / Approach
                                                                                                         39%            21%
The second centers on the phases of flight that were                         / Landing
most prone to error. The third question queries about                                                                Insufficient
                                                                             Taxi / Park                  2%
the most frequent types of error and their                                                                               data
consequences.      The final question asks whether
airlines are subjected to the same distribution of error                   There was also a large number of errors in the
in their operations.                                                     preflight / taxi phase. They were typically associated
                                                                         with checklist usage, documentation, and incorrect
What is the extent of flightcrew errors in normal                        switch and lever settings.        In addition, error
flight operations?
                                                                            The definition for a consequential outcome is a flightcrew error
   It has long been argued that pilot error is an                        either linked to undesired aircraft state or another error. An
everyday occurrence but to what degree? The LOSA                         undesired aircraft state is defined as an outcome in which an aircraft
results presented in Table 5 offer some initial insight.                 is placed in a compromising situation that poses an increased risk to
                                                                         safety. These can include altitude deviations, unstable approaches,
                                                                         and heading deviations.

Klinect, J.R., Wilhelm, J.A., & Helmreich, R.L. (1999). Threat and error management: Data from line operations safety audits.
  In Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 683-688). Columbus, OH: The Ohio State University.

management during the phase was substantially better                        In order to elicit more detail from flightcrew errors,
than the descent / approach / landing phase with 7% of                   content domains were derived across the five error
the errors becoming consequential.                                       types4. These results show that 31% of the 578 errors
                                                                         in the LOSA database were associated with
What are the most frequent types of errors and                           automation (MCP and/or FMC errors). This was
which are the most consequential?                                        closely followed by checklist errors (24%), sterile
                                                                         cockpit errors (13%), ATC related errors (8%) and
   As specified in the model of flightcrew error                         briefing errors (6%). The automation and checklist
management, there are five types of error: (1)                           error domains need to be examined in more detail.
intentional noncompliance (otherwise known as
willful     violations),    (2)    procedural,     (3)                      Of the 179 automation errors, 21% were associated
communication3, (4) proficiency, and (5) operational                     with incorrect switch settings or execution of modes.
decision errors (Helmreich, Klinect, and Wilhelm, this                   These are the typical slips or lapses with automation
volume). A distribution of error types and the                           (Reason, 1997). However, the majority of automation
percentage that were linked to consequential outcomes                    errors (65%) observed were associated with a failure
are presented in Table 7.                                                to cross-verify settings. As a whole, 11% of the
                                                                         automation errors were linked to consequential
         Table 7 – Distribution of Error Types and                       outcomes.
                  Consequential Outcomes
                                           Percentage                       In the checklist error domain, 7% of the 139 errors
                       Percentage of
       Error type                           that were                    were linked to an additional error or undesired aircraft
                                          consequential                  state. The most frequent checklist error was the
    Intentional                                                          performance of a checklist from memory (37% of the
                            54%                2%                        total). Others included the lack of challenge and
    Procedural              29%               23%                        response to a checklist (26%), checklist completed but
                                                                         not called “complete” (22%), and omitting or
    Communication            6%               13%
                                                                         performing the wrong checklist (8%). The remaining
    Proficiency              5%               69%                        7% were slips and lapses at the item level (e.g., missed
    Operational                                                          item).
                             6%               43%
                                                                         Do all airlines have the same distribution of errors?
   The most frequent type of error, intentional
noncompliance, was also the least consequential. The                        Similar to external threats, there were significant
two percent that were consequential were intentional                     differences between airlines in the number of errors
violations of checklist procedures, such as performing                   committed and the percentage of errors that became
a checklist from memory. All of these willful                            consequential (Table 8)5.
checklist errors were linked to incorrect lever and
switch settings. This concurs with previous research                                 Table 8 – Error Variability by Airline
that states that violations are only of consequence                                                Airline     Airline     Airline
when they are paired with an unintentional error (Free,                                                A           B         C
1994; Hudson, 1998).                                                         Average errors
                                                                                                      .86         1.9       2.5
                                                                             per flight segment
   While proficiency and operational decision errors                         Percentage of
were the least often observed, results indicate that they                    errors that were        18%         25%        7%
were also the most difficult for flightcrews to manage.                      consequential
When both of these types were combined into one
category, they were linked to consequential outcomes                        There were also differences in the types of error
over 70% of the time. Future research will investigate                   committed across airlines. For example, Airline C had
further into these types of error in the hope of learning                a significantly higher proportion of intentional
what flightcrews are doing in their mismanagement.                       noncompliance errors. This is an illustration of how
                                                                         the proactive measurement of flightcrew errors allows
                                                                         airlines to uncover possible conditions in an airline
                                                                         that facilitates thew commission of certain errors.
  Current research is integrating models of threat and error
management with content coding and linguistic analyses, in an
investigation of cockpit communication processes (Sexton &                   Error content domains are in the appendix (Table 11).
Helmreich, this volume)                                                      Chi squares were significant at .05 level

Klinect, J.R., Wilhelm, J.A., & Helmreich, R.L. (1999). Threat and error management: Data from line operations safety audits.
  In Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 683-688). Columbus, OH: The Ohio State University.

This surprisingly high number of intentional                             of behaviors and management strategies. This will be
noncompliance errors might indicate the need for an                      a hypothesis for future research.
airline to evaluate procedures, manuals, or the check
airman program to ensure greater compliance.                             What are the typical outcomes of error?

               ERROR MANAGEMENT                                            The results show that 85% of the flightcrew errors
                                                                         were inconsequential. However, 15% of the errors
   The notion of error management has steadily gained                    observed were linked to an additional error or
momentum in the world of pilot training. In the                          undesired state (Table 10).
simplest form, error management consists of an error,
flightcrew response, and outcome. Using this logic, a                    Table 10 - Distribution of Flightcrew Error Outcomes
number of questions will be addressed in the                                  Error Outcome           Percentage of Outcomes
discussion of the error management results. The first
                                                                          Inconsequential                              85%
two questions ask about the typical responses and
outcomes to error. The next question touches on the                       Undesired Aircraft State                     12%
extent of undesired aircraft states and how are they                      Additional Error                              3%
managed during flight. Lastly, the final question asks
about CRM behaviors that were cited as the most                          What is the extent of undesired aircraft states and
important in error management.                                           how are they managed?
What are the typical responses to error?                                    Undesired aircraft states are theorized to be at the
                                                                         cusp of an incident or accident. Out of the 314 flight
   When an error is committed, there are three types of                  segments, 67 undesired states were observed. The
responses: trap, exacerbate, and no response. For a                      most frequently observed were vertical deviations
trap response, the error is detected by the flightcrew                   from the assigned altitude (16% of the undesired
and actively managed to an inconsequential outcome.                      states), aircraft speed above limits (12%), and unstable
On the other hand, an exacerbated error is one that is                   approaches (10%).
detected, but mismanaged in a way such that it induces
additional errors or undesired states. The final                            As for flightcrew responses to undesired aircraft
response is a failure to respond to an error. The                        states, 79% were mitigated. An example of a
outcomes for these types of errors can either be                         mitigated response is when a flightcrew violates an
inconsequential or linked to undesired aircraft states                   altitude and immediately flies back to the assigned
and additional errors. To get an more accurate picture                   altitude. However, not all undesired aircraft states are
of error management, the analysis is only going to                       effectively managed. Approximately nine percent
look at flightcrew responses to procedural,                              were exacerbated into flightcrew errors.            The
communication, proficiency, and operational decision                     remaining 12% were states in which the flightcrew did
errors. This is done because these types of error are                    not respond and allowed the state to resolve itself. An
unintentional and thus more likely to elicit an active                   example of undesired aircraft states include long
error management response (Table 9).                                     landings and landings outside of the touchdown zone.
  Table 9 - Distribution of Flightcrew Responses to                      What behaviors were observed to be the most
                 Unintentional Errors                                    important in error management?
                                    Percentage of
      Error response
                                 unintentional errors                       Expert observers were asked to list behaviors that
 Trap                                   36%                              they thought were important in the flightcrew
 Exacerbate                             11%                              management of error. The most frequent behaviors
 No Response                            53%                              listed were vigilance, crewmembers speaking up and
                                                                         stating pertinent information, and crewmembers
   The majority of the error responses were a failure to                 asking questions regarding crew actions and decisions.
respond. Interestingly, 36% of these errors were linked
to undesired aircraft states. When an undesired                                                  SUMMARY
aircraft state occurs, the flightcrew no longer manages
the error. Instead, they are managing an the potential                     This paper presented a unique perspective on threat
consequences of an undesired state. This subtle                          and error management. This insight into normal flight
difference may lead flightcrews to utilize distinct sets                 operations allowed us to learn the following points.

Klinect, J.R., Wilhelm, J.A., & Helmreich, R.L. (1999). Threat and error management: Data from line operations safety audits.
  In Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 683-688). Columbus, OH: The Ohio State University.

                                                                         error management. The University of Texas Team
    External threats and flightcrew errors are                          Research Project Technical Report 99-01.
     pervasive in normal flight operations but can
     differ in their frequency and type across airlines                       Helmreich, R. L., Klinect, J. R. & Wilhelm, J. A.
                                                                         (this volume). Models of threat, error, and response in
    The descent / approach / landing phase of flight                    flight operations. In Jensen, R. S. (Ed.), Proceedings
     contained the most threats, errors, and                             of the tenth international symposium on aviation
     consequential outcomes.                                             psychology, Columbus, OH: Ohio State University

    Intentional noncompliance errors were the most                          Hudson, P.T.W. (1998) Bending the Rules:
     frequently committed and also the least                             Violation in the Workplace. Invited Keynote Address,
     consequential.     Proficiency and operational                      Society of Petroleum Engineers, 1998 International
     decision errors were the most difficult for                         Conference on Health, Safety and Environment in Oil
     flightcrews to manage.                                              and Gas Exploration and Production. Caracas. June
    The most common errors committed were
     associated with automation and checklists. The                           Hines, W. H. (1998). Team and technology:
     majority of these errors were not the typical slips                 Flightcrew performance in standard and automated
     or lapses, but a failure to cross verify settings or                aircraft. Unpublished doctoral dissertation. The
     incorrect usage.                                                    University of Texas at Austin.

    For the most part, flightcrews failed to respond to                     Reason, J. T. (1997).            Managing the risks of
     errors after they were committed. Most of the                       organizational accidents.              Aldershot, United
     time they were inconsequential but there was a                      Kingdom: Ashgate.
     significant amount that turned into undesired
     aircraft states in which flightcrew were very                            Sexton, J. B. & Helmreich, R. L. (this volume).
     successful in mitigating their consequences.                        Analyzing cockpit communication: The links between
                                                                         language, performance, error, and workload. In
    Many CRM behaviors were cited in effective /                        Jensen, R. S. (Ed.), Proceedings of the tenth
     ineffective threat and error management. These                      international symposium on aviation psychology,
     included leadership, vigilance, stating important                   Columbus, OH: Ohio State University
     information, asking questions, and contingency
     planning.                                                           University of Texas Crew Research Project Website:
   These lessons are only a start in our understanding
of safety in normal operations. Future research will                                              APPENDIX
test theories across aircraft types (automated vs.
conventional), crew positions, and other demographics                               Table 11 - Content Error Domains
like crew experience. In addition, the linkage between                    Sterile cockpit errors      Lever or switch errors
CRM behaviors and threat and error management will                                                    (not automation)
                                                                          Callout errors
be further established. By presenting these results, it
                                                                          ATC related errors               Radio errors
was our hope to stimulate new forms of thought about
how we view aviation safety.                                              Checklist errors                 Documentation errors
                                                                          FMC/MCP errors                   Air and ground
                                                                                                           navigation errors
                                                                          Hard warning errors
     Free. R.J. (1994). The Role of Procedural                                                             Crew to crew
Violations in Railway Accidents. Unpublished Ph.D.                        Briefing errors
                                                                                                           communication errors
thesis, University of Manchester.                                         Proficiency errors
                                                                                                           Miscellaneous errors.
     Helmreich, R. L., Klinect, J. R., Wilhelm, J. A., &                  Decision errors
Jones, S. G. (1999). The Line/LOS Checklist, Version
6.0: A checklist for human factors skills assessment, a
log for external threats, and a worksheet for flightcrew


Lingjuan Ma Lingjuan Ma