Troubleshooting Plant Process Upsets

							Equifactor® and Process Troubleshooting:
Managing Your Crucial Tribal Knowledge

Jason G. Laws, P.E.

Jason G. Laws, PE
    



Engineering Sales Worked in Research Project Engineering Pilot Plant Engineering Plant Engineering and Engineering Management Plant Management

Introduction


Loss of Institutional Knowledge across the industry
– –

Globalization, Cultural Shifts and Outsourcing Contributes to Incidents Years of process upsets and recovery experience. I looked around and I was the veteran.



My Personal Experience
– –

  

What Do we do? Example Walk through the Equifactor Maintenance Editor

Process Knowledge Critical


KAIZEN: The Key to Japan‟s Competitive Success, Masaaki Imai.
–

–
–

Many Engineers work far from the Process Wanted best and senior talent involved in Production. Experienced Cross functional Teams critical for success.

Wide Spread Loss of Knowledge
 

Tenured engineers & operators are ready to retire.
– – –

Will lose10 to 20% of senior technical personnel in 3 to 4 years. (De Long) “A crisis of experience will occur within five years” De Long 5-10 yr Problems, process upsets, hazards.


Haven‟t made up the talent along the way.
Existing operators crews may never have had a turnaround.



This leaves a serious brain drain on the whole industry.
–

Paul Loftus writes the shortage is a function:
  

An ageing workforce The image of industry with young people And their desire for mobility in their career.



Without an industry wide effort there will be a crisis!
–

Judith Hackett warns that industry must heed these warnings.

The Culture of Tenure
  

Talent entering the work force wants career mobility. They don‟t see staying with one company Will there be enough experience to ever make up the loss?
– –

Tribal Institutional Knowledge was prevalent. This is fading away quickly.
Retirees leave and succeeded by inexperienced talent.

 

Creates a draft of novices to replace the void?
–

Who will be around to see 5 to 10 year problems.
–
–

Where will the solutions be then? This is why we are here!

Supporting Documentation


Chemicals Industry Leaders: Are you Ready for the Workforce of the Future?, David De Long How to Tackle Talent Management, Paul Loftus. Amnesia Can Kill, Judith Hackitt.

 




Where are chemical engineers headed? , Mark Rosenzweig.
Better Practices for Retaining Organizational Knowledge: Lessons From the Leading Edge ,David De Long. Avoid a Dumb Move: Stinting on capturing the expertise of departing employees is a false economy Mark Rosenzweig. Companies Keep Know-ho in Place, Sean Ottewell





What can be done?
 

Get in front of the Problem! Work it on two fronts.
1. Encourage:
  

Retention of new employees. Expand efforts to retain high value senior talent as a gap measure! New hires to absorb knowledge until the inevitable occurs!

2. Develop KNOWLEDGE TRANSFER PROGRAMS.
 

Create them NOW! (De Long). Equifactor® can serve as a tool.

My Personal Experience


I had several mentors along the way.
– –

We worked in teams in a small Plant. Slowly many of my mentors advanced away or retired.



I found myself promoted away from daily involvement in the Plant.
–

I was managing new Process Engineers



The new guys had lot‟s of supervision leadership experience, but:
– – –

Little hazardous process knowledge. No oxide experience. The V.P of Manufacturing was on record saying Supervisors and Mangers didn‟t need strong process knowledge. That came from operations and controls investment.



Soon I was the one of the most experienced leader in the Plant and Company.

What About Process Upsets?
 

Quality Issues and Process Upsets?
–

600 Products, 14 process units, 7 basic process families. Some are seen only every few years. Related to Product Family and Process Knowledge.
  

Many Process Upsets come up over the years.
– –



Learn from.
Seeing them occur. Trouble Shooting and RCA. War stories of the company sages.

Examples of Process Upsets and Causal Factors
   

Product gels, why?
–

Using acrylic acid exotherm to heat up the batch.

Hetero. Cat. Batch won‟t kick (initiate), why?
–

Charging solvent with a dirty hose.
The agitator on R-XX5 set too low for family. Tube leak in a heat exchanger.

Batch too viscous, why?
–

Batch has low hydroxyl, why?
–

Equifactor® & the Hines Bloch Checklist.
 

I was in my first Equifactor® class.
–

Ken Reed Introduced the Customizable Check Lists. Could we use them for Process Upsets? Retiring a Plant Chemist and my eventual absence? A particular process upsets underlined the critical need. Equifactor® Check Lists ideal for capturing this knowledge.

The Customized Check Lists.
–

–
– –



This began the dialogue that brought us here today.

I Returned from Vacation.


I overhear the Plant Chemist and the Production Superintendent talking about two bad batches.
– –

They had low hydroxyl numbers. Several things can lead to the quality issue. Everything seemed okay, but the Batch was still bad.

  

The Plant Chemist had checked the QC and was perplexed.
–

Third Batch Planned! I asked I the seal was leaking or if there was a tube leak on the heat exchanger.
– –

Water in the batch can cause the low hydroxyl. I reflected on the compartmentalization of knowledge even with tenured talent.
  

We each had our favorite usual suspects. That is where the check list comes in. I recently had a „usual suspect‟ moment the other day. I have to remind myself not to jump to conclusions.

First Day Back for The Plant Engineer

1/2/2008 1/2/2008 7:36 AM

2 Batches in a Row have Gone Bad.
Sr. Plant Engineer returns from 2 wk Vacation Sr. Plant Engineer hears Plant Chemist talking about the 2 Batches in the hallway Sr. Plant Engineer asks if the seal pot on the pump is leaking

2nd Base Catalyzed Polymer Batch goes bad.

Sr. Plant Engineer Goes to the Reactor Plant

QC shows low Hydroxyl

Sr. Plant Engineer and Process Engineer begin both investigations.

The Cooling Tower Foaming
1/2/2008 7:30 AM 1/2/2008 8:31 AM

Lead Operator is performing AVO route

Cooling tower is foaming over.

Lead Operator contacts the Process Engineer

Process Engineer Calls the Sr. Plant Engineer and Maintenance Dept.

Process Engineer goes to the Utilities Area

Chemical Control Failure the Usual Suspect

Equifactor Process
Using Equifactor®
Spring SnapCharT® 1 Plan Investigation – Get Started! Root Cause Tree® Equifactor® Summer SnapCharT® Equifactor® CHAP

?? ?? ?? ??
Keep Process Check Lists under the Equipment Tables

2

Determine Sequence of Events

Change Analysis 3 Define Causal Factors Autumn SnapCharT® Equifactor® Safeguards Analysis 4 Analyze Each Causal Factor's Root Causes Root Cause Tree®

5

Analyze Each Root Cause's Generic Causes

Root Cause Tree® Corrective Action Helper® Corrective Action Helper® SMARTER Matrix Safeguards Analysis

6

Develop & Evaluate Corrective Actions

7

Present / Report & Implement Corrective Actions

Winter SnapCharT® TapRooT® Software

The Process Check List

Base Catalyzed Polyol Problem: Low Hydroxyl

•Low/Wrong Monomer •Low Catalyst •Water Intrusion •To Much Solvent

Possible Causes

Usual Culprits to Check!

Incident Timeline
1/2/2008

2nd Base Catalyzed Polymer Batch goes bad.

1/2/2008 7:36 AM

Sr. Plant Engineer returns from 2 wk Vacation

Sr. Plant Engineer hears Plant Chemist talking about the 2 Batches in the hallway

A

Production Supervisor Requests making another Batch to meet orders

Sr. Plant Eng goes to Control Room to check data historian.

Sr. Plant Engineer and Process Engineer check pump and Agitator Seal Pots for leaks.

Fall Snap Chart® : Low Hydroxyl
1/2/2008 1/2/2008 7:36 AM

2nd Base Catalyzed Polymer Batch goes bad.

Sr. Plant Engineer returns from 2 wk Vacation

Sr. Plant Engineer hears Plant Chemist talking about the 2 Batches in the hallway

Sr. Plant Engineer asks if the seal pot on the pump is leaking

Sr. Plant Engineer Prints out Process Upset Check List from Equifator.

A

QC shows low Hydroxyl

Plant Chemist confrims Catalyst Level

Cooling Tower is Foaming.

?

Plant chemist confirms NVR

Reactor Isolated from CT by Tempered Water HEX

Fall SnapChart ® Continued.

B

Production Reques another B ord

?

Water From Wall Manifold.

Condenser Agitator Seal Pot Scrubber/Flare

Sources of Water 1. 2. 3. 4. 5. 6. Overhead Condenser Scrubber/Flare Line Agitator Seal Pot. Pump Seal Pot Tempered Water HEX shell. Water from Manifold.
Pump Seal Pot Tempered Water Shell

Fall SnapChart ® Continued.
C Operations and Plant Engineering Check the Tubes for Leaks SPE suspects H20 and tries to find another source At a loss SPE orders the manway opened on the reactor. Ops and Eng visually confrims water pool in reactor and drip from top of vessel SPE requests Ops bump the agitator. D

No Leaks from the tubes

Other Possible Sources Identified.

?

Drip isn't near any nozzle.

Leak increases in flow rate.

Overhead Condser has water. Not Used during this batch.

Water is charged to reactor with hoses from external manifold.

Would need to have hose hooked up and leak by 4 block valves.

Would need to leak across tubes and closed valve. Could explain foam in the CT.

Fall SnapChart ® Continued.

The Seal Pot

Cooling Tower Water to Seal Jacket
Pumping ring

Barrier Fluid

Royal Purple Synthetic

Water Isolated from Process By Stainless Jacket
Inboard Seal Face Cooling Jacket for Seal

Cooling Tower water plumbed to pumping Ring.

Barrier Fluid

Barrier Fluid in the Cooling Jacket

Roll it Into the Root Cause Tree.

TapRooT® Root Cause Tree®
START HERE

Causal Factor/Issue:

with eac h c aus al factor/iss ue and s elec t or eliminate each c ategory to find root causes.

HUMAN PERFORMANCE DIFFICUL TY

EQUIPMEN T DIFFICUL TY*

Pag es 5 -21 *No te : Consider Equifactor™ Ana lysis

NATURAL DISASTER / SABOTAG E

O THER (SPECIFY)

TOL ERABLE FAIL URE DESIGN SPECS

DESIGN

EQUIPMEN / T PARTS DEFECTIVE DESIGN REVIEW PROCUREMENT

PREVENTIVE / PREDICTIVE MAINTENANCE PM NI

REPEAT FAILURE MANAG EMENT SYSTEM Corr ective Actio n

MANUFACTURING Spe cs NI Design Not To Spe cs Pro blem Not Anticipated eq uipment en vir onment no t consider ed Ind epe nde nt Review NI ma nag emen t of chan ge (moc) NI ha zard a nalysis NI HANDLING STORAGE QUAL ITY CONTROL No PM for Equ ip PM fo r Equ ip NI

corr ective actio n NI corr ective actio n no t yet impleme nted tren ding NI

Human Performance Human Performance Troubleshooting Troubleshooting Guide Guide
(15 Questions) (15 Questions)
Directions: Dire ctions: Answer a questions Answer all ll qu estio ns an then refer to the and d th en refer to the ind ica te Basic Cause indicated d Ba sic Ca use Catego ries o the back Ca tegories on n th e back of th sheet to invesof thisis sheet to in ve stigate the ca uses o tigate the causes of f the pr oble m. the problem.

Indiv idual Individua l Pe rforma nc e Performance Pag es 2 2-31 Pages 22-31

Basic Cau se Catego ries Basic Cause Categories (o n back) to investigate (on back) to investigate

NINI NEEDS IMIMPROVEMENT = = NEEDS PROVEM ENT Ma y also substitute LTA (L ess Than Ad equ ate) Ma y also substitute LTA (L ess Th an Adequate) or or PIO (Potential Improvement Opportunity) PIO ( Pote ntial Impr oveme nt O ppor tu nity) Te a m Team Pe rforma nc e Performance Basic Cau se Catego ries Basic Cause Categories (o n back) to to investigate (on back) investigate

Was p erson excesWas aaperson excessive ly fatigued impaired, sively fatigued, , imp aire d, up se bor ed, d istra cted upset,t,bored, distracted or o ve rwhelmed? or overwhelmed?

Yes Yes

HUMAN ENGINEERING HUMAN E NGINEERING WORK DIRECTION WORK DIRECTION WORK DIRECTION WORK DIRECTION PROCEDURES PROCEDURES HUMAN ENGINEERING HUMAN E NGINEERING

Pag es 3 1-33 Pages 31-33 Did ve rba l communication s Did verbal communicationsYes Yes COMMUNICATIONS or shift chan ge pla y a ro le or shift change play a role CO MMUNICATIONS in this pproblem? in this roble m? Did fa ilure to agr ee ab out Did failure to agree about the who /wha t/whe n/wher e the who/what/when/where of per fo rming the jo b play of performing the job play a ro le in th is pro blem? a role in this problem? Yes Yes TRAINING TRAINING

Sho uld the per so n have Should the person have ha d and u sed a wr itten had and used a written pr ocedu re but did no t? procedure but did not?

Yes Yes

COMMUNICATIONS CO MMUNICATIONS WORK DIRECTION WORK DIRECTION

Was a mistake mad e Was a mistake made while u sin g a p roced ure ? wh ile using a procedure?

Yes Yes

PROCEDURES PROCEDURES

Were a larms or displa ys to Were alarms or displays to re co gnize or to re sp ond to recognize or to respond to a cond itio n u navaila ble or a condition unavailable or misun der stood ? misunderstood?

Yes Yes

Was communication Was communication ne ede d across or gan iza- Yes needed across organizaYes COMMUNICATIONS tiona l b ound arie s or with CO MMUNICATIONS tional boundaries or with othe r fa cilities? other facilities?

HUMAN ENGINEERING HUMAN E NGINEERING Basic Cau se Catego ries M anage me nt Basic Cause Categories Ma nagement (o n back) to investig ate Syste m (on back) to investigate System Pag es 3 3-35 Pages 33-35 Yes Yes MANAG EMENT SYSTEM MANAGEMENT SYSTEM Was a task pe rformed in a Was a task performed in a hu rry or a sh ortcut u se d? WORK DIRECTION hurry or a shortcut used? WORK DIRECTION Had mana geme nt b een Had management been Yes Yes MANAG EMENT SYSTEM war ned o f th is pr oble m or MANAGEMENT SYSTEM warned of this problem or ha d it ha ppen ed be fo re? had it happened before? Were p olicie s, admin. Were policies, admin. contro ls, or p roced ure s controls, or procedures no t used , missin g, o r in not used, missing, or in ne ed of impro ve men t? need of improvement? Yes Yes MANAG EMENT SYSTEM MANAGEMENT S YSTEM WORK DIRECTION WORK DIRECTION PROCEDURES PROCEDURES

Yes Yes Were d isp lays, a larms, co ntrols, Were displays, alarms, controls, tools, o equip men ide ntifie tools, orrequipment tidentifiedd or or op era te d impro per ly? operated improperly? Did th e person n eed more skill/ Yes Did the person need more skill/ knowle dge to per fo rm the job o r toYes knowledge to perform the job or to re sp ond to co ndition s or to un derrespond to conditions or to understa nd system respo nse? stand system response?

TRAINING TRAINING PROCEDURES PROCEDURES HUMAN ENGINEERING HUMAN E NGINEERING TRAINING TRAINING WORK DIRECTION WORK DIRECTION

Was wo rk pe rformed in an Was work performed in an ad ve rse environ men t (such as adverse environment (such as ho t, humid, da rk, cramped , or hot, humid, dark, cramped, or ha zard ous)? hazardous)?

Yes Yes HUMAN ENGINEERING HUMAN E NGINEERING

Revise d 9/30/99 Re vised 3/20/00 Copyrig ht © 1 999 b y Co pyright © 2000 by System Improvements, Inc. System Improvements, Inc. Kno xville , Tenn esse e, USA Knoxville, TRe se rved USA All Rig hts ennessee, All Rightstion Prohibit e d Duplica Reserved Duplic ation P rohibite d

Did work invo lve r epe titive Did work involve repetitive Yes mo tio n, u nco mfortab le Yes mo tion, uncomfortable po sition s, vibr ation, or positions, vibration, or he avy liftin g? heavy lifting?

TRAINING TRAINING HUMAN ENGINEERING HUMAN E NGINEERING WORK DIRECTION WORK DIRECTION

Sho uld an ind epe nde nt Yes Should an independent YesQUAL ITY CONTROL qu ality control check quality control check QUALITY CONTROL ha ve cau ght the p rob lem? have caught the problem? Continu e on back b y an alyzing the indicated Ba sic Ca use Categ orie s Co ntinue on back by analyzing the indicated Basic Cause Categories

Winter SnapChart ®

Corrective Actions

•Labels NI •Arrangement Placement •Plant Unit Differences

•No Procedure
•Task Not Analyzed

Conclusion Part 1.

   

Over all loss of process knowledge IS taking place. Who is going to solve our problems in the future? Need methods of knowledge transfer. Lead to an Economic loss just on 1.5 days.
–

~ $250K.



Equifactor Trouble Shooting.

Customizable Check Lists: Part 2.
  

Open the Database Review Some Processes Create a New Process
– – – –

Process Possible Cause Symptoms Remedies

View Database Info.

Possible Causes and Trouble Shooting Check List

Creating a New Process


Add a New Process
– –

ACS Filtration Add in Equipment on the same level.



Add a Symptom
–
–

Filtration Pump Shuts Down Add Below the Process Level.
Reactor Level Low. Check Reactor Level and Start New Batch.



Add Possible Causes
–



Add Check and Solution.
–

Open Equifactor Maintenance

Click on Top Process

Click New to Start New Process

ADD A PROCESS

New Process Click on Same Level

Click OK to Save

Type Process Description

Click New

Click on ACS Filtration to Add a Symptom

ADD A SYMPTOM

Click on Sub-level
Click OK

Enter Symptom Description

Click to Add Possible Cause

Then Click New

ADD A POSSIBLE CAUSE

Click Sub-Level Click OK

Add Cause Description

Add a Remedy

Use Sub-level

Click OK

Descriptions, instructions and warnings.

Fill in with Process Details, pictures etc.

Conclusion: Part 2.


Any Process Can be Mapped with the Tables.
– – –

Sales, Customer Service and Quality etc. all have process. Incidents occur because of system break downs. Not just for Safety Incidents or Mechanical Failures.

  

Capturing process knowledge easy with Equifactor® Equifactor® can play a role in knowledge management. The coming crisis makes this is CRUCIAL!