507 by pengxuebo


									LOSA: Line Operations Safety
  Audit: History and Status

       Robert L. Helmreich, PhD, FRAeS

         Human Factors Research Project
         The University of Texas at Austin

                  November, 2003
    Data to Isolate Safety Issues
   Accident investigation
       Limited, non-representative sample
   Incident reports (CHIRP, ASRS and ASAP)
       Data slanted to events resulting from system and flight crew failures
       UT HF developing data category system with AA & CO
   Formal checkrides (Line and Proficiency)
       Data show crew capability and procedural knowledge
   Flight Data Recorders – QAR (FOQA)
       Data show “what happened” in terms of flight parameters
   Non-jeopardy observation of normal flights-LOSA
       Give data on why things happen and how they are managed
       Provides realistic baseline of safety data
       Is proactive
   Jump seat observations of flight crew performance during
    regular scheduled flights

   Observers unobtrusive – collecting data not
    participating in flight
       Team of observers from different backgrounds
               Line pilots / Union representatives
               Check airmen
               Safety and Training pilots
               Credible outside observers (for reliability check)

            Scientific Background
   LOSA uses systematic observation of behavior
   Systematic observation is a validated
    methodology using observers trained and
    calibrated to high reliability
   Systematic observation has been employed in
    scientific studies of crew performance in
    demanding environments
       For example, Aquanauts in undersea habitats during
        Project SEALAB and Project Tektite
            History of LOSA
   Formal LOSA developed at request of Delta
    Airlines to validate operational impact of
    Human Factors (CRM) training

   The focus of initial LOSA was systematic
    assessment of CRM-related crew behaviors

   In 1997, collaboration with Continental
    Airlines to expand LOSA to include threats
    and errors and their management
                Purpose of LOSA
   Provide valid empirical data on:
      1. Crew performance – strengths and
             Proficiency
             Decision-Making
             CRM skills
             Procedural compliance
             Threat and error management
     2.   System performance – strengths and
             Culture
             Airspace System – airports and navigational Aids
             Aircraft design / automation
             Standards / Training / Safety / Maintenance
             Crew support – ATC, Cabin, Ground, and Dispatch
    Threat and Error Management
         LOSA (1997-2003)
•   Continental Latin     •   EVA Air
    America               •   Uni Air (Taiwan)
•   Continental Express   •   Frontier
•   Air New Zealand       •   QANTAS
•   Air Micronesia        •   Singapore
•   Continental           •   Silk Air
                          •    Braathens
•   Delta
                          •   Alaska
•   USAirways
                          •   Air New Zealand (2nd)
•   Cathay Pacific
     LOSA Components
   Part 1. Threat prevalence & mgt
   Part 2. Error prevalence & mgt
   Part 3. CRM Countermeasures
   Part 4. Flight crew interview
   Part 5. Flight crew survey
         LOSA Data: Flight Crew
For each flight segment, observers collect data on:
General Flight Crew Information   Flight Crew Performance
   Demographics                     Behavioral markers
   Attitudes / perceptions /        Crew errors and
    safety culture (FMAQ)             violations
   Safety interview
                                     Undesired aircraft states
                                     Technical data for
Flight Description
   Observer narrative
                                         Type and stability
   Overt threats
                                     Threat management
   Operational complexity
                                     Error management
                                     Undesired state
                 Threat Categories
• Threats - Originate outside the flight crew’s influence but require active
  management to prevent them from becoming consequential to safety

  Environmental Threats                        Airline Threats
    Adverse weather                     Operational time pressure
    ATC events / errors                 Cabin events / errors
    Terrain                             MX events / errors
    Traffic                             A/C malfunctions / MELs
    Airport conditions                  Ground / Ramp events /
                                         Dispatch events / errors
                                         Ground crew events / errors
                Latent Threats
   Aspects of the system that predispose the
    commission of errors or can lead to undesired
    aircraft states
       ATC practices
       Organizational, national, professional culture
       Aircraft characteristics
       Qualification standards
       Regulatory practices and oversight
       Flawed procedures
       Scheduling and rostering practices
                     Threat Prevalence
Last 10 LOSAs
   1835 flights with 7576 threats
        4.1 threats per flight average

   98% of flight segments had one or more threats
        Range across airlines – 94% to 100%

   2/3 were environmental threats; 1/3 were airline threats
        44% of environmental threats occur in descent / approach / land
        72% of airline threats occur in predeparture

   Most prevalent threats
        Adverse weather – 26% of all threats
        ATC clearances/late changes - 21% of all threats
                  Threat Management
Last 10 LOSAs
   960 mismanaged threats in 1835 flights
        Average 1 mismanaged threat every 2 flights
   87% of threats effectively managed
        Range across airlines – 79% to 92%
   Environmental and airline threats managed equally well
   Highest mismanagement rates
        17% of ATC threats were mismanaged
        15% of Aircraft malfunctions /MELs threats were mismanaged
                Error Avoidance

   Complete error avoidance is impossible –
    errors are inevitable
   80%+ of all flights observed had one or
    more errors
       Must look for sources of error to strengthen
        system defenses
      Broad Error Categories

Intentional Noncompliance – violations
      ex) Performing a checklist from memory
Procedural – Followed procedures but wrong
      ex) Wrong altitude setting dialed into the MCP
Communication – Missing information or
   misinterpretation within cockpit
      ex) Miscommunication by crew with ATC
Decision – Discretionary decision that unnecessarily
   increases risk
      ex) Unnecessary navigation through adverse weather
            Decision Error

Choice increasing risk in a situation with
   multiple courses of action possible
   time available to evaluate alternatives

   no discussion of consequences of alternate
   courses of action
   no formal procedures to follow
                   Error Prevalence

Last 10 LOSAs
 1835 flights with 5172 errors = 2.8 errors per
  flight on average
 82% of flight segments had one or more errors
        Range across airlines – 70% to 94%
   32% of all errors are intentional noncompliance
              Error Response

   25% Detected and action taken

   30% Detected and ignored
       Almost intentional noncompliance

   45% Fail to detect
       Red Flag: improve monitoring and cross-checking
                   Error Outcomes
   71% Inconsequential
   21% Undesired aircraft state
    8% Additional error

   1484 mismanaged errors in 1835 flights = 0.8
    mismanaged errors per flight on average

   77% of intentional noncompliance errors were
       Are procedures in need of review?
           Undesired Aircraft State
A compromised situation placing the flight at increased
  •   Lateral deviation                  •   Unstable approach

  •   Vertical deviation                 •   Abrupt aircraft control

  •   Speed too high                     •   Long landing – no go

  •   Speed too low                          around

  •   Incorrect aircraft configuration   •   Firm landing
       –   Flight controls
                                         •   Forced landing
       –   Systems
       –   Fuel                          •   Wrong taxiway, ramp,
       –   Automation
                                             runway, country

                                         •   Runway incursion
        Error/Outcome Summary





                 0        20       40          60         80   100

                     % of errors        % consequential
     7% of errors involved a lack of technical proficiency
            Phase of Flight Effects
                                Threats by         Errors by
      Phase of Flight
                                  Phase             Phase
     Pre-Departure / Taxi          40%               26%
       Takeoff / Climb             15%               20%
            Cruise                 8%                6%
    Descent / Approach /
                                   33%               44%
          Taxi / Park               4%                4%

   Descent / approach / land phase contains the most variability
    in crew performance and consequential errors
                Threat and Error
   Is a high level of threat associated with
    more errors?
       There are slightly more unintentional errors
        when there are many threats, but the
        relationship is weak
       There is no relationship between threats and
        intentional non-compliance
                   Profile of Airline X
                                       Average from 4 LOSA
Major Statistics           Airline X    Archive Comparison
Average number of
threats per flight           4.3              4.3
% of threats that were
mismanaged                  8%               15%
Average number of
errors per flight            3.0              2.5
% of errors that were
mismanaged                  18%              31%
                         Airline X
                                       Average from 4 LOSA
Major Statistics           Airline X    Archive Comparison
Average number of
mismanaged errors per        56                75
100 flights
Average number of
undesired states per         42                53
100 flights
Average number of
mismanaged undesired          6                8
states per 100 flights
    Myths from the Days of Jurassic
   Myth 1: The glass cockpit will eliminate
    human error on the flight deck
       Reality: Automation is the second largest
        source of pilot error

   Myth 2: Dog and Duck automation will
       3 individuals in crew
       Pilot, dog, duck
          Pilot reaches for controls
          Duck quacks

          Dog wakes and bites pilot

       Reality: the human is still in the loop
      FMAQ Safety Scale Items
   My suggestions about safety would be
    acted upon if I expressed them to

   Management will never compromise safety
    concerns for profitability

   I am encouraged by my supervisors and
    coworkers to report any unsafe conditions
High scores on the FMAQ Safety
Scale are associated with fewer
  violations of SOPs and fewer
    Undesired Aircraft States
             University of Texas
        Threat and Error Management
             Model (UT-TEMM)
   The Threat and Error Management model was derived
    from LOSA data and guides further refinement.

   The model is being used by airlines as a framework
    for analysis of incident and ASAP data

   IATA is using it as the framework for analysis of
    worldwide accidents and incidents
Threat and Error Management Model
                      Latent and Overt

                     Threat M anagement


   Inconsequential   Error M anagement    Incident / Accident

                     Undesired Aircraft

                     Undesired Aircraft
                     State M anagement
           Applying LOSA Data
LOSA data have two primary uses:
1. Assessing system safety
2. Identifying issues for action
      LOSA database has data on airports, aircraft, crew
       experience, organizational and professional culture
      Providing airlines with feedback on their own
      Observers provide valid record of what crews do on the
      Show areas of strength as well as those needing
      Data help airlines prioritize and evaluate safety efforts
          CRM training
           Using Error Data for
        Organizational Interventions
   Violations - suggest poor procedures, weak captain
    leadership and/or a culture of non-compliance

   Procedural errors - may indicate poor workload
    management and/or poor procedures

   Communications errors - may reflect inadequate CRM
    (monitoring and challenging) or complacency

   Decision errors - may indicate need for more CRM
    training on expert decision making and risk assessment
              LOSA and CRM

   CRM has evolved through 6 generations

   LOSA data provide airlines with critical
    topics for 6th generation CRM with its
    focus on threat and error management
    Emerging Problems with CRM
   Not accepted by all (boomerangs)

   Purpose not understood -- “to help us get
    along better”

   Lacking a universal rationale that can be
    accepted in every culture and organization

   Slippage over time – need for refresher
          Fifth Generation CRM :
         Error Management 1996-
   Focus on managing human error
   Training in limitations of human
       Universal nature of human error
       use of incident & accident data to illustrate
   Continuation of earlier generation training
    topics under error management framework
Continental Airlines used LOSA data
   to develop Threat and Error
    Management CRM training
       – the 6th Generation
       Sixth Generation CRM:
    Threat and Error Management
               1999 -
   Fifth generation was useful, but created
    resistance among pilots who did not like
    the idea that their task was managing
    their own errors
   LOSA documents not only errors but also
    the nature and prevalence of threats in
    the operating environment – shows
    superior performance as well as problems
           Training Issues in Threat
            and Error Management
   Human limitations as sources of error
   The nature of error and error management
   Culture and communications
   Expert decision-making
   Training in using specific behaviors and procedures as
    countermeasures against threat and error
    –   Briefings
    –   Inquiry
    –   Sharing mental models
    –   Conflict resolution
    –   Fatigue and alertness management
   Analysis of positive and negative events and accidents
        Significant LOSA Events

   LOSA Weeks
     Hong Kong
     Panama

     Dubai

     Dublin

   ICAO Journal May 2002 features LOSA
   ICAO LOSA Handbook 2002
Recognition and Extension of LOSA

Flight Safety Foundation/Aviation Week and Space
   Technology Laurels, 2001
DOSA - Dispatch LOSA (with Continental)
University of Texas Center of Excellence in Patient
   Safety adopting LOSA methodology for
   systematic observational research into operating
   rooms and emergency rooms
          The LOSA Collaborative
    An umbrella research organization led by James Klinect
Collaborative functions
   1)   Implementing LOSA and maintaining the integrity of the
        process and database by certifying observers and data
   2)   Supporting research efforts of the University of Texas
        Human Factors Research Project
   3)   Offering user interface data to manufacturers to enhance
        design process and de-identified system data to industry
        and regulators
   4)   Coordinating a support group of participating LOSA airlines
        to exchange information on data use and oversight for the
    The University of Texas
 Human Factors Research Project


To top