Lies, Damned Lies, and Statistics Hanford Performance Indicator Forum December 11, 2003 Steven S Prevette Fluor Hanford Safety and Health 509-373-9371 Steven_s_prevette@rl.gov http://www.hanford.gov/safety/vpp/trend.htm Introduction • Numbers abound in the Safety profession • We have all seen great uses and misuses • How can we make the best use of the data we have and make effective decisions that will improve the safety of ourselves, our coworkers, and our companies? Liars Figure and Figures Lie 1 18 16 Injuries by Month 14 12 10 8 6 4 2 0 Jan-02 Jul-02 Jan-03 Jul-03 Jan-04 Apr-02 Oct-02 Apr-03 Oct-03 Here are the past 2 years of injury data. What is your interpretation? What is your prediction of future rates? Liars Figure and Figures Lie 2 18 16 Injuries by Month 14 12 10 8 6 4 2 y = -0.132x + 11.987 0 Jan-02 Jan-03 Jan-04 Jul-02 Jul-03 Apr-02 Oct-02 Apr-03 Let’s add a ―trend line‖. What is your Oct-03 interpretation now? What is your prediction of future rates? Liars Figure and Figures Lie 3 18 Injuries by Month 16 14 12 10 8 6 4 2 6 Month Moving Average 0 Jan-02 Jan-03 Jan-04 Jul-02 Jul-03 Apr-02 Oct-02 Apr-03 Oct-03 Let’s add a 6 month moving average. What is your interpretation now? What is your prediction of future rates? Interlude – Fun with Charts • It appears I can make this data say anything I want it to say . . . • Courses are taught on ―chartsmanship‖ • Can we interpret the data with a sound, rigorous method that is repeatable and gives credible results? Liars Figure and Figures Lie 4 25 Average = 10.3 (Jan02 - Jan04) 20 Upper Control Limit 15 10 5 Lower Control Limit 0 Nov-02 Nov-03 Jan-02 Jan-03 Jan-04 Jul-02 Jul-03 May-02 Sep-02 May-03 Sep-03 Mar-02 How about the Control Chart? Mar-03 - The data are stable, within certain bounds. The Red Bead Experiment • Many of you have experienced the ―Red Bead Experiment‖ • Results changed from person to person, but were random and predictable • The data here were generated by the Red Beads • Which chart gave the proper interpretation and predictions? The Control Chart • Separates random noise from signal • Provides prediction capability • When coupled with management theory and action can lead to significant improvements • Does not require great costs to perform • Actually has saved money Management Theory For Improvement • When there is a trend, find ―special cause‖ and act upon the trend • When there is no trend, find ―common causes‖ and act upon the system, the overall performance Pareto Charts OSHA Cases FY 1997 Fluor Hanford OSHA Cases FY 1998 through May 1998 "Overall 25% Reduction to Date" 0.40 0.35 0.30 0.25 0.20 0.15 -48% -26% -8% 72% 0.10 -63% 0.05 0.00 OR R HP T GE R IAN R AT TT E RIG RIC OPE E FI E R/ ECT S PIP RK EL CES WO RO IRO N ARP CLE NU Pareto charts can be used to analyze common causes – FH ―Top 5 Occupation‖ initiative from 1998 What is a Trend? • One point outside the control limits • Two out of Three points two standard deviations above/below average • Four out of Five points one standard deviation above/below average • Seven points in a row all above/below average • Ten out of Eleven points above/below average • Seven points all increasing/decreasing. See http://www.hanford.gov/safety/vpp/trend.htm Leading Indicators • This system can be used with leading indicators and Behavior Based Safety • A personal example – my group targeted computer ergonomics • Goal was to improve behaviors on the computer, to reduce risk of Carpal Tunnel / Overuse Syndrome Leading Indicator Example BEHAVIOR TO OBSERVE YES NO Backrest of chair is supporting person's back Forearms are parallel to floor Knees at angle of 90 - 110 degrees Legs supported by chair seat Person's eyes are 18 - 30 inches from screen Top of monitor at or below eye level Monitor viewed straight ahead with no neck turn Wrists are in proper position (Wrist Support in use, or wrists held at level they would be if there was a Wrist Support) Typing stand (document holder) at same height and distance as screen [leave blank if no stand in use] No glare of outside light, overhead lights, or work lights on screen. Leading Indicator Results • Initial rate was 10% safe behaviors • Plotted on Control Chart • Pareto chart of individual items established • Safe behavior increased to 83% within 5 months, a significant increase Barriers to Good Data Use • Higher ups will use it as a ―hammer‖ • Subjected to quotas and targets imposed from above • Fear (―accountability‖) used as a ―motivator‖ • Actions and Explanations as a result of random fluctuations • Perceived loss of control over portrayal of performance The Biggest Barrier = FEAR Data Sources • Worker and Customer Opinion • Expert Review • Process Measures – Top-Down (start with mission/vision) – Process Approach – Bottom-Up (start with available data) Data Sources THEORY Dollars Process Hours OUTCOMES Materials Cycle Time Product Mission Progress Data Service Commitments Met In-Process Stewardship Budget Inventory Schedule Profit Safety Procedure Satisfaction Compliance Rework Idle Time Waste More Fun with Numbers • Boston Herald (Dec 23, 1996): "Murders Sink To 30-Year Low " • "....major crime such as homicide is down. Over the last two years there has been a 52% reduction in shootings..... that's a dramatic difference....Police Commissioner Paul Evans has succeeded where others have fell short...." More Fun with Numbers 2 Homicides per Year, Boston MA Average = 88.5 160 (1976 - 1989) 140 Number of Homicides c-chart UCL 120 100 80 60 c-chart LCL New Chief 40 in 1993 20 0 1964 1966 1968 1970 1972 1974 1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 Year “Murder rate rises in Boston” BOSTON (AP) — The number of murders in Boston rose in 2000 after years of steady decline, partially due to an alarmingly violent first six months. The final tally was 37 — up from 31 last year. [Police Commissioner] Evans noted that there were no homicides in October or November For 2001, AP reported a 67% increase Conclusion – Data Sanity We can either react to numbers, with explanations of every percent change, with the inherent frustrations, fear, and failure Or We can understand our data, put it to good use, and apply valid management principles The choice is ours.