Sensitivity Analysis of Subjective Ergonomic Assessment Tools

Sensitivity Analysis of Subjective Ergonomic Assessment Tools Impact of Input Information Accuracy on Output (Final Scores) Generation Occupational Safety & Ergonomics Program Industrial & Systems Engineering Department Sensitivity Analysis of Subjective Ergonomic Assessment Tools: Claudia P. Escobar Thesis Committee • Dr. • Dr. • Dr. • Dr. Jerry Davis, Chairman Robert Thomas Saeed Maghsoodloo Nathan Dorris Claudia P. Escobar Industrial & Systems Engineering Department Contents 1. 2. 3. 4. 5. 6. 7. 8. 9. Literature Review Identified Gap Objective Methodology Results Limitations Further Research Self-Reporting Conclusions Claudia P. Escobar Industrial & Systems Engineering Department Literature Review • Risk factors such as force, posture, movement, vibration, etc. are thought to directly increase the risk for musculoskeletal disorders. (Li & Buckle, 1999) • The validity of an ergonomic assessment tool depends on the level of accuracy an evaluator can achieve when assigning scores to these factors. (Faragasanu & Kumar, 2002) Claudia P. Escobar Industrial & Systems Engineering Department Literature Review • Key factors to select an appropriate ergonomic assessment tool: – – – – – – – – – Ease to use Training level for evaluator Applicability of the results Economic issues Time constraints Equipment required Work disruption Need for a data analyst Usability (adequacy and validity) (Waters, Putz-Anderson, and Baron, 1997; Waters, Baron, and Kemmlert, 1998) Claudia P. Escobar Industrial & Systems Engineering Department Literature Review • Ergonomic assessment tools can be subjective or objective in nature. Subjective tools are more predisposed to evaluator’s bias. (Faragasanu & Kumar, 2002) • Self-reporting provides valuable insight into working conditions, and is a low cost, low risk, cost effective method. (Marley & Kumar, 1996; Woodcock, 1986; Ramsay, 1993; Andrews, Norman, & Wells, 1996; Faragasanu & Kumar, 2002) Claudia P. Escobar Industrial & Systems Engineering Department Literature Review • Self-reporting may be biased and have low validity/reliability in relation to the needs of the assessment. (Jacobs, 1998; Li & Buckle, 1999) • The level of subjectivity directly affects the tool’s validity and reliability. The higher the reliability, the greater the strength and confidence. (Faragasanu & Kumar, 2002) Claudia P. Escobar Industrial & Systems Engineering Department Identified Gap It is required to provide a tool that ensures validity during the input information collection: – By means of offering input variables discriminated in categories easily distinguishable. – With values that the observer can compare with those existing in the assessed job, and select without making a mistake in the process. Claudia P. Escobar Industrial & Systems Engineering Department Identified Gap • Risk factors such as force, frequency and duration can be assessed without major difficulties. • Posture-based conditions require subjective estimations that may result in biased and inaccurate classifications. • Mistakes could be made more frequently when assessing these stressors. Claudia P. Escobar Industrial & Systems Engineering Department Identified Gap This investigation was derived from the detected need of evaluating the levels of accuracy required when collecting information for input posture-based variables. Claudia P. Escobar Industrial & Systems Engineering Department Identified Gap • There are no studies ranking the importance of input variables when considering validity of outcomes. • Only JSI offers one of its input variables as the most critical. Claudia P. Escobar Industrial & Systems Engineering Department Objective To determine the effects input posture-based variables have on the final hazard level classification, when using subjective ergonomic assessment tools, by means of sensitivity analysis. Claudia P. Escobar Industrial & Systems Engineering Department Objective (Specific) To detect the non-sensitive, sensitive, and critical input posturebased variables for three subjective ergonomic assessment tools. Claudia P. Escobar Industrial & Systems Engineering Department Methodology 1. 2. 3. 4. 5. Tool pre-selection Selection criteria Tools selected Sensitivity analysis Critical variables identification Claudia P. Escobar Industrial & Systems Engineering Department Tool Pre-selection • Fifteen tools were pre-selected according to their self-reporting applicability. • Described in terms of main objective, input/output information, limitations, validity, reliability, and sensitivity. Claudia P. Escobar Industrial & Systems Engineering Department Tools Evaluated 1. 2. 3. 4. Revised NIOSH Lifting Equation Rapid Upper Limb Assessment (RULA) Rapid Entire Body Assessment (REBA) Ovako Working Posture Analysis System (OWAS) Posture, Activity, Tools and Handling (PATH) Liberty Mutual Tables for Lifting, Carrying, Pushing and Pulling (Snook Tables) Job Strain Index (JSI) ACGIH TLV for Hand Activity Claudia P. Escobar Industrial & Systems Engineering Department 5. 6. 7. 8. Tools Evaluated 9. 10. 11. 12. 13. 14. 15. ACGIH for Work-Related Musculoskeletal Disorders Screening Tool for Lifting Rodgers Muscle Fatigue Analysis Borg Scales of Perceived Exertion OSHA Screening Tool – VDT Checklist WISHA Lifting Analysis WISHA Hand-Arm Vibration Analysis WISHA Checklist Claudia P. Escobar Industrial & Systems Engineering Department Tools Evaluated (REBA) Description Assesses levels of risk on sedentary workers. The method uses diagrams of body postures and three scoring tables to provide the evaluation. RULA is based on OWAS. Analysis in the saggital plane. Validity, Accuracy, Limitations Validated for a few types of jobs, such as computer users and sewing machine operators. It requires no special equipment and can be done in confined spaces without workforce disruption. Confounding factors are worker’s age and experience, workplace environment, and psychosocial variables. Risk Factors Evaluated Force. Repetition. Awkward postures. Areas of Body Addressed Arm. Wrist. Neck. Trunk. Leg. Input Information Required Most sensitive to postures of arm, wrist, elbow, neck and trunk. Output Information Obtained Upper and lower arm position. Wrist position and twisting. Duration for the adoption of the posture. Load weight. Force/load frequency. Neck position. Trunk position. Leg position. Sensitivity Score for hazard exposure level, which may vary from 1 to 7. Self-Reporting Almost all the variables required as input information can be self-reported. Variables such as load weight can be obtained as exact measures. Claudia P. Escobar Industrial & Systems Engineering Department Selection Criteria 1. Input and output data used: – Quantitative – Qualitative 2. Type of assessment yielded: – Objective – Subjective 3. Self-reporting potential 4. Focus of the tool’s variables – Posture-based Claudia P. Escobar Industrial & Systems Engineering Department Tools Selected Rapid Upper Limb Assessment (RULA) (McAtamney & Corlett, 1993) Rapid Entire Body Assessment (REBA) (Hignett & McAtamney, 2000) Job Strain Index (JSI) (Moore & Garg, 1995) Claudia P. Escobar Industrial & Systems Engineering Department RULA    Claudia P. Escobar Industrial & Systems Engineering Department RULA • RULA is one of the most popular ergonomic assessment tools in industry. • User-friendly. • Only an initial estimation is required. No major calculations needed. • Perfectly matches the selection criteria for the study. Claudia P. Escobar Industrial & Systems Engineering Department REBA      Claudia P. Escobar Industrial & Systems Engineering Department REBA • REBA follows the same principles as RULA. • Used for both static and dynamic postures. • User-friendly. • Uses tables to compute scores. • Perfectly matches the selection criteria. Claudia P. Escobar Industrial & Systems Engineering Department JSI Intensity of Exertion Crit Light Somewhat hard Hard Very hard Near maximal Val 1 3 6 9 13 Duration of Exertion Crit < 10 10-29 30-49 50-79 ≥ 80 Val 0.5 1 1.5 2 3 Efforts/ Minute Crit <4 4-8 9-14 15-19 ≥ 20 Val 0.5 1 1.5 2 3 Hand/Wrist Posture Crit Very good Good Fair Bad Very bad Val 1 1 1.5 2 3 Speed of Work Crit Very slow Slow Fair Fast Very fast Val 1 1 1 1.5 2 Duration per Day Crit ≤1 1-2 2-4 4-8 ≥8 Val 0.25 0.5 0.75 1 1.5 Claudia P. Escobar Industrial & Systems Engineering Department JSI • JSI focuses on hand and wrist conditions. • Obtained from the product of the six multipliers. • Did not absolutely match the selection criteria. • Wide applicable and popular. It has been validated. Claudia P. Escobar Industrial & Systems Engineering Department Sensitivity Analysis 1. Creation of data sets (combinations) 2. Correlation analysis (Pearson’s test) 3. Non-sensitive variable identification 4. Brute force method and simple linear regression 5. Critical variables identification Claudia P. Escobar Industrial & Systems Engineering Department Creation of Data Sets • Created iterating simultaneously each input variable within its range of values (combination). • Final scores and final hazard level classifications identified for each combination. • Only posture-based variables included. • Modifiers were excluded from the iterations. Claudia P. Escobar Industrial & Systems Engineering Department Data Set (example) 4 2 3 3 2 3 2 4 4 5 5 5  Claudia P. Escobar Industrial & Systems Engineering Department Data Sets • RULA  10,368 combinations. • REBA  2,160 combinations. • JSI  7,500 combinations. Claudia P. Escobar Industrial & Systems Engineering Department Correlation Test Variable Upper arm Lower arm Wrist Wrist twist Variable Neck Trunk Legs Score A 0.88 0.15 0.29 0.09 Score B 0.74 0.56 0.11 RULA Score Score A Score B Score C 0.62 0.67 Claudia P. Escobar Industrial & Systems Engineering Department Correlation Test Variable Trunk Neck Legs Variable Upper arm Lower arm Wrist Score A 0.71 0.40 0.56 Score B 0.94 0.17 0.25 REBA Score Score A Score B Score C 0.80 0.56 Claudia P. Escobar Industrial & Systems Engineering Department Correlation Test JSI Variable Intensity of exertion JSI 0.40 Hand/wrist posture Duration of exertion Efforts/minute Speed of work 0.32 0.32 0.23 0.16 Duration per day 0.32 Claudia P. Escobar Industrial & Systems Engineering Department Correlation Test (Results) • All variables were found sensitive. • Sensitive variable  has any kind of influence in the final hazard level classification. Claudia P. Escobar Industrial & Systems Engineering Department Sensitivity Analysis 1. Brute Force Method (RULA and REBA) 2. Simple Linear Regression Model (JSI) Claudia P. Escobar Industrial & Systems Engineering Department Brute Force Method • Simple, straight-forward method. • Individual disturbance of discrete inputs while the rest remains constant. • Uses a base case (expected values). • Applied to RULA’s and REBA’s data sets. Claudia P. Escobar Industrial & Systems Engineering Department Base Case Calculation Score 1-2 2 3 Total 1 3-4 2 3 48 33 106 45 67 71 Value 1 Freq. 25 % 25/106 = 23.58 45.28 31.13 100.00 24.59 36.61 38.80 Example: RULA’s Wrist Value 1 Frequency 25*23.58+45*24.59 +63*29.30+44*28.0 3 = 47.75 % 47.75/232.46 = 20.54 2 3 Total 107.81 76.90 232.46 46.38 33.08 100.00 Total 1 5-6 2 3 Total 1 183 63 70 82 215 44 78 35 157 2 3 100.00 29.30 32.56 38.14 100.00 28.03 49.68 22.29 100.00 Expected value for wrist (rounded) 1*20.54% + 2*46.38% + 3*33.08% = 2 7-higher Total Claudia P. Escobar Industrial & Systems Engineering Department Base Case RULA Variable Upper arm Lower arm Wrist Wrist twist Neck Trunk Legs Score A Score B Score C REBA 4 2 3 2 3 3 2 4 5 5 Variable Trunk Neck Legs Upper arm Expected Value 3 2 2 4 Expected Value Lower arm Wrist Score A Score B 2 2 5 6 Score C 7 Claudia P. Escobar Industrial & Systems Engineering Department RULA RULA's Upper Arm 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 Extreme postures Change from 5-6 to 7-higher Scores Final Score Base Case < 45° to > 45° Values Claudia P. Escobar Industrial & Systems Engineering Department Change from 3-4 to 5-6 REBA REBA's Trunk < 20° to > 20° 12 11 10 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 Change from 4-7 to 8-10 Final Score Base Case Scores Values Claudia P. Escobar Industrial & Systems Engineering Department Critical Variable With its change from a specific value to the next, it produces an increase (or decrease) in the hazard level classification. Claudia P. Escobar Industrial & Systems Engineering Department Critical Variables REBA: • • • • • Trunk Neck Legs Upper arm Wrist RULA: • • • • Upper arm Neck Trunk Legs Claudia P. Escobar Industrial & Systems Engineering Department Critical Variables REBA is more prone to a linear behavior when disturbing critical variables than RULA. Claudia P. Escobar Industrial & Systems Engineering Department Ranking Variable Upper arm Lower arm Wrist Wrist twist Variable Neck Trunk Legs Score A 0.88 0.15 0.29 0.09 Score B 0.74 0.56 0.11 RULA Score Score A Score B Score C 0.62 0.67 Claudia P. Escobar Industrial & Systems Engineering Department Ranking RULA: 1. Upper arm 2. Neck 3. Trunk 4. Legs Claudia P. Escobar Industrial & Systems Engineering Department Ranking Variable Trunk Neck Legs Variable Upper arm Lower arm Wrist Score A 0.71 0.40 0.56 Score B 0.94 0.17 0.25 REBA Score Score A Score B Score C 0.80 0.56 Claudia P. Escobar Industrial & Systems Engineering Department Ranking REBA: 1. Trunk 2. Upper arm 3. Legs 4. Neck 5. Wrist Claudia P. Escobar Industrial & Systems Engineering Department Results If the posture is near the base case, only the critical variables will directly change the final hazard classification. Claudia P. Escobar Industrial & Systems Engineering Department Results - RULA Upper Arm: • Shoulder flexion from <45 to >45. • Added shoulder raised and/or upper arm abduction. Neck: • Neutral posture to >10 flexion. Claudia P. Escobar Industrial & Systems Engineering Department Results - RULA Trunk: • Flexion change from <20 to >20. Legs: • Any misclassification. Claudia P. Escobar Industrial & Systems Engineering Department Results - REBA Trunk: • Change from neutral to >20. • Change from <20 to >20 extension. Neck: • Added twist or tilt-to-side conditions. Legs: • Change in knee flexion from <60 to >60. Claudia P. Escobar Industrial & Systems Engineering Department Results - REBA Upper arm: • Added shoulder raised and/or arm abduction/rotation conditions. Wrist: • Added wrist twist/deviation conditions. Claudia P. Escobar Industrial & Systems Engineering Department Simple Linear Regression Model • For each variable, a coefficient is computed. • The smaller the coefficient, the greater the influence on final scores. Claudia P. Escobar Industrial & Systems Engineering Department Simple Linear Regression Model JSI = 5.761 IE + 23.04 (DE + EM) + 19.66 HWP + 24.58 SW + 46.08 DD – 184.32 R2 = 54.30% Claudia P. Escobar Industrial & Systems Engineering Department Ranking Intensity of exertion Speed of work Hand/wrist posture Duration of exertion and efforts per minute 5. Duration per day 1. 2. 3. 4. Claudia P. Escobar Industrial & Systems Engineering Department Analysis of the Results 1. Preliminary/complimentary studies. 2. Conclusions 3. Limitations 4. Future research 5. Self-reporting 6. Final conclusions Claudia P. Escobar Industrial & Systems Engineering Department Preliminary/Complimentary Studies 1. Grouped variables analysis for RULA and REBA.  additive effects.  180,000 combinations for RULA.  55,000 combinations for REBA. 2. Simple linear regression model with more degrees of freedom.  same R2. Claudia P. Escobar Industrial & Systems Engineering Department Conclusions • It is inaccurate to assume that all input variables are equivalent in influence on outcomes. • Focus on RULA and REBA should start with upper arm and trunk posture assessment, respectively. • Focus on JSI should start on intensity of exertion estimation. Claudia P. Escobar Industrial & Systems Engineering Department Conclusions – RULA / REBA • An increment in final hazard level classification was often found when additional awkward conditions were added. • The more awkward the posture was found, the more sensitive to changes the tool was. Claudia P. Escobar Industrial & Systems Engineering Department Conclusions • The greatest proportions of combinations from data sets described jobs with high levels of hazards. • It is difficult to find a “safe” job. Claudia P. Escobar Industrial & Systems Engineering Department Conclusions RULA: •  48%  5-6 •  32%  7-higher •  20%  lowest REBA: •  44%  4-7 •  43%  8-10 •  11%  lowest •  2%  highest JSI: •  24%  safe •  76%  risk Safe jobs! • RULA  0.81% • REBA  1.3% • JSI  24% Claudia P. Escobar Industrial & Systems Engineering Department Conclusions If a medium or high hazardous job is detected, and improvements are performed, are the tools going to provide information that would help determine if such improvements were adequate? Claudia P. Escobar Industrial & Systems Engineering Department Conclusions If a company wants to evaluate the working conditions for its workers, and uses RULA, REBA, or JSI, is it ever going to find results reflecting a safe work environment? Job =  Tasks Claudia P. Escobar Industrial & Systems Engineering Department Conclusions • Perhaps, before analyzing the tool’s validity, it would be appropriate to study the tool’s approach. • A too conservative approach could eliminate the possibility of detecting minor changes and improvements in working conditions. Claudia P. Escobar Industrial & Systems Engineering Department Limitations • More techniques for sensitivity analysis could be used. • More tools must be analyzed. • Sensitivity analysis applied not only to expected values but also to minimum, maximum, and random working conditions. Claudia P. Escobar Industrial & Systems Engineering Department Self-Reporting • The results of the study are useful when trying to evaluate how appropriate self-reporting would be if used during an intervention. • Training for self-reporter should target critical variables. Claudia P. Escobar Industrial & Systems Engineering Department Future Research • Extend the study to more ergonomic assessment tools. • Expand and modify the selection criteria used. • Include other working scenarios (extreme and random conditions). Claudia P. Escobar Industrial & Systems Engineering Department Final Conclusions • The study provides the best results possible, considering its scope and limitations. • Because it is known which variables cause the most impact on hazard level determination, methods to ensure accuracy and validity during their assessment can be successfully developed and implemented. Claudia P. Escobar Industrial & Systems Engineering Department Final Conclusions • Levels of training should target the critical variables identified. • The results of the study provide a valid and strong reference to focus the subjective component that potentially dominate the hazard level outcome. Claudia P. Escobar Industrial & Systems Engineering Department Acknowledgements Dr. Jerry Davis Dr. Robert Thomas Dr. Saeed Maghsoodloo Dr. Nathan Dorris Michael Gray Family and Friends Claudia P. Escobar Industrial & Systems Engineering Department Thank You ! Questions? Claudia P. Escobar Industrial & Systems Engineering Department Sensitivity Analysis of Subjective Ergonomic Assessment Tools