Learning Center
Plans & pricing Sign in
Sign Out



									PEP-II Operational Safety
   and Management

      Presentation by Roger Erickson

   B-Factory Operations Review
             SLAC, April 26, 2006

              Accelerator Matrix Management
              Communications with Stakeholders
              MCC Control Room Staffing
              Operator Profile and Training
              Accelerator Operations Documentation
              Maintenance Management
              ROD and Down Time Planning
              Conclusions

04/26/06                                  B-Factory Operations Review
           Main Control Center

04/26/06                         B-Factory Operations Review
                             Matrix Management
           Accelerator programs are managed through a matrix organization
           structure with links to:
          Program Coordinators for PEP-II and Linac Pulsed Programs.
          Liaison persons representing each experiment.
          System Physicists for each major section of the accelerator facility.
          Area Managers – one for each major section, who report to the
           System Physicists.
          Accelerator physicists and engineers responsible for subsystems or
           commissioning tasks.
          Technical support departments with engineering and technician
           support for on-call maintenance.
          Accelerator Division Safety Office.
          ES&H Departments, including Radiation Physics and OHP.

04/26/06                                                          B-Factory Operations Review
                     Communications with Stakeholders
          8:00 Meeting every day, seven days a week, in MCC conference room.
            •   Operations report by EOIC, including run time statistics, progress in carrying out
                scheduled program, descriptions of problems, etc.
            •   Reports by representatives of major experimental user groups.
            •   Attended by PPA, Accel Div, and Operations management reps, Program Deputy,
                System Physicists, Area Managers, and representatives of user groups and major
                technical support departments; typical attendance: 45 persons.

          8:15 Maintenance Meeting, five days each week.
            •   Run by Accelerator Operations Department maintenance managers.
            •   Attended by supervisors of all maintenance groups, Area Managers, and interested
                physicists and engineers.
            •   Attendance: typically 45 persons.

          Accelerator Plans Meeting, weekly.
            •   Review of operational progress, problems, and plans.
            •   Attended by Accel Systems Div and Dept Heads, Program Coordinators, Accelerator
                Section Heads and ADSO.
04/26/06                                                                          B-Factory Operations Review
                               Control Room Staffing
          Program Deputy
           •   Duty rotates among Accelerator Division physicists.
           •   Oversees program for 3 or 4 day period.
           •   Not always present in the control room.

          Operations Engineer
           •   1 EOIC per shift (from a group of 7), on rotating 7-week cycle.
           •   Responsible for safely carrying out the scheduled program.
           •   Exempt position; directs the Accelerator Systems Operators.
           •   Must be present if accelerator is capable of producing beam.
           •   All have experience as accelerator systems operators.

          Accelerator Systems Operators
           Typically 3 operators per shift on rotating 5-week cycle:
           • Linac and sources.
           • PEP-II rings.
           • Secondary program, PPS control, and backup.

04/26/06                                                                         B-Factory Operations Review
                   Typical Operator Profile
           Typical applicant is a recent graduate with a
           bachelor’s degree in physics.

               •   13 have bachelor’s degrees.
               •   7 have master’s degrees.
               •   4 have a technical background and extensive SLAC

               •  Accelerator Systems Operator 1
                   • 1-2 years of experience

               •  Accelerator Systems Operator 2
                   • 2-5 years of experience

               •  Accelerator Systems Operator 3
                   • 5+ years of experience

               •  Operations Engineer
                   • 5+ years of experience

04/26/06                                                   B-Factory Operations Review
                         Operator Training Program
          Workbook and evaluation for each classification:
            •   Accelerator Systems Operator 1
            •   Accelerator Systems Operator 2
            •   Accelerator Systems Operator 3
            •   Operations Engineer

          ES&H Training (including RWT-II and LRCA)

          Accelerator Operations Safety Procedures Training
           (including PPS operation, searching, interlock testing for each zone.)

          US Particle Accelerator School

          Downtime seminars and special projects

04/26/06                                                                B-Factory Operations Review
                   Accelerator Operations Documentation

                                  Accelerator Division Operations Directives

               Administrative Documents                               Safety Documents
             example: Administrative Guidelines                        Search Procedures
                        (11 titles)                                        (45 titles)

           Accelerator Operations Documents                          Training Documents
                  Linac Turn-on Checklist                         ASO-2 Qualification Workbook
                         (38 titles)                                      (17 titles)

               Maintenance Documents                              Documentation Documents
                        Call-in Lists                             AD Documentation Archive Plan
                         (4 titles)                                        (5 titles)

                                        Archived Documents and Logs
                                             MCC Operations Log
                                                 (83 titles)

04/26/06                                                                                B-Factory Operations Review
                AD Safety Documents with Site-Wide Applicability

          SLAC Guidelines for Operations
          Work Authorization for the Linac, PEP-II, FFTB, and Associated
          Summary of Requirements for Work in SLAC Accelerator
          Restart Plan for the Linac, PEP-II, and BABAR
          Restart Plan for the FFTB and ESA Accelerator Programs as an
           Extension of the Linac, PEP-II, and BABAR Restart Plan
          Linear Accelerator Facility Safety Assessment Document
          PEP-II Safety Assessment Document
          BaBar Safety Assessment Document
          Policy for Working with Electrical Hazards in SLAC Accelerator
          Lock and Tag Verification Procedures for Accelerator Magnets in
           the Linac Complex and PEP-II

04/26/06                                                      B-Factory Operations Review
                   Accelerator Operations Safety Documents
          MCC Key Safe Checkout Log                Safety Inspection Checklists
          CID PPS Log                              PPS Interlock Checklists
          DRIP/DR PPS Log                          Beam Containment System
          Linac PPS Log                             Procedures
          Positron PPS Log                         High-Voltage Test Facility Safety
          BSY PPS Log                               Procedures
          ESA PPS Log                              BSOIC Certification Checklists
          FFTB PPS Log                             Gun Safety Procedures
          PEP PPS Log                              Laser Safety Procedures
          PLS Keys Procedures and Log              Fire Alarm System Reference Manual
          MCC Spare Keys Key Safe Checkout         PEP PPS Overview
          PPS Zone Maps                            Safety Configuration Work Control
          Logging Requirement for Work on          Incident Response Procedures
           BCS Devices                              Alarm Response Procedures
          VVS Remote Turn-on Procedure             Warning Response Procedures
          Search Procedures                        Accelerator Operations BAS
          Electrical Hazard Test Procedures         Instructions
          Entry and Exit Procedures                Accelerator Operations BAS Key and
                                                     Lock Procedures

       •163 controlled documents in the Accelerator Division (incl. non-safety related).
       •MCC Operations Log is now electronic, web-accessible, and archived.
04/26/06                                                                  B-Factory Operations Review
                          Operator Safety Procedures
           Operators are trained to carry out certain well-defined tasks,
           following formally controlled procedures.
            •   Interlock Checklists
            •   Search Procedures
            •   Entry and Exit Procedures
            •   Safety Inspection Checklists

           Safety procedures:
            •   undergo a formal review and approval process.
            •   are updated as conditions change.
            •   are available on the AD Documentation web site, although the signed
                originals are preserved in paper form.

           Completed checklists and log books are archived.

04/26/06                                                              B-Factory Operations Review
                            Maintenance Management
          Problems that interrupt the scheduled program trigger immediate repairs
           to restore operations.

          24-hour on-site staff to repair:
            •   Mechanical utilities
            •   Power supplies
            •   Modulators
            •   Cryogenics

          On-call staff for:
            •   Electrical power
            •   Magnets and other mechanical systems
            •   RF systems
            •   Vacuum systems
            •   OHP

          Extensive call-in lists identify first responders, alternates, and experts for
           all systems.

04/26/06                                                                  B-Factory Operations Review
                           Opportunistic Maintenance
          Opportunistic Approach
            •   Whenever possible, identified maintenance, repair, and installation jobs are pre-
                planned, but deferred until an opportunity arises. The accelerator systems are
                not turned off for maintenance on a routine, scheduled basis.

          Coordination
            •   Daily 8:15am meeting.
            •   Coordinated by Operations Maintenance Office.
            •   Area Managers are advocates for their areas.

          Standby Maintenance List (SML)
            •   Jobs that are planned and can be launched on short notice.
            •   Provides options for EOIC to take advantage of unscheduled interruptions.

          Repair Opportunity Day (ROD)
            •   Launched only when required by a driving event.
            •   Carefully planned, and typically involve 100 or more jobs.

          ARTEMIS Electronic Database

04/26/06                                                                       B-Factory Operations Review
          ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance
           Information System) is an Oracle database with a Remedy interface,
           accessible over the internet by everyone involved with SLAC
           accelerator maintenance.
          Operators report problems as they occur by making entries in
          Repair work is launched immediately if the problem is inhibiting the
           scheduled accelerator program.
          New ARTEMIS entries are reviewed at the daily maintenance meeting.
           Open questions are addressed and jobs are assigned.
          Jobs may be assigned to the Standby Maintenance List (SML), or to the
           Repair Opportunity Day (ROD) list, or marked for the next long down
          Jobs are tracked using ARTEMIS until they are finally closed.

04/26/06                                                          B-Factory Operations Review
           Daily Maintenance Meeting

04/26/06                               B-Factory Operations Review
           Review of New Problems
           Sample from an actual maintenance meeting.

04/26/06                                                B-Factory Operations Review
           Standby Maintenance List

04/26/06                              B-Factory Operations Review
                       Repair Opportunity Days
          A ROD list of pending repair jobs is derived from the
           ARTEMIS database and maintained by the Operations
           Department. Installation and PM jobs are added as needed.

          All ROD jobs are reviewed in the daily maintenance

          The ROD list can be viewed over the internet.

          When a “driving event” occurs, a detailed plan can be
           assembled and launched quickly, using the ROD list.

          A typical ROD day includes about 100 distinct jobs.

04/26/06                                                   B-Factory Operations Review
                          Long Down Time Planning
     Long Down Times usually occur once each year and typically last two to three months.

     Area Managers collect job requests from system physicists, project engineers, maintenance
      groups, and the ARTEMIS database.

     A comprehensive schedule is prepared, updated, and reviewed at weekly meetings, starting
      about four months before the down time.

     This schedule lists each job with an assigned shop, and lists the Project or Group Manager,
      Responsible Task Person, and Area Manager.

     Schedule specifies time and duration of each job (with 1 day time resolution), and number
      of persons required.

     Schedule specifies authorizations required (WAF, RSWCF, EWP, etc.).

     Detailed (shift by shift) schedule of pre-run testing, safety certifications, and turn-on
      activities is prepared by the Operations Dept and revised as needed on a real-time basis.

04/26/06                                                                     B-Factory Operations Review
                                 Maintenance Statistics
          ARTEMIS entries since Jan 1989: 77,328                       (as of April 18, 2006)

          Total identified problems not yet closed: 456
           Many of these involve non-critical problems that were observed, but went away without a
           clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance.

          ARTEMIS entries in one year (since April 18, 2005):
            •   3166 new problems (8.7 per day).
            •   450 problems resulted in lost PEP-II run time.
            •   3296 problems closed (9.0 per day).

          Active jobs:
            •   SML: 52
            •   Next ROD: 32
            •   Aug-Nov down time: 80

04/26/06                                                                           B-Factory Operations Review
                          Operational Safety Reviews

          Linac and PEP-II Re-Start Validation Review (March 2005)

          FFTB Re-Start Review (June 2, 2005)

          ESA Re-Start Review (July 19, 2005, secondary beams only)

          Director’s Integrated Safety Management (ISMS) Review (Sept 2005)

          DOE ISMS Review (October 2005)

          Injector Test Facility Re-Start Validation Review (October 2005)

          ESHAC Review (November 2005)

          ESHAC Review (March 2006)

04/26/06                                                                B-Factory Operations Review
                   New Safety Processes Introduced in FY05
          JHAMs (Job Hazard Analysis and Mitigation) individualized and
           documented for every employee.

          Expanded training requirements for most workers.

          AHAs (Area Hazard Analysis) posted for every building.

          Arc-flash analysis and warning labels for circuit breakers and
           contactors (except Cat 0 light switches, etc.)

          All hoisting and rigging equipment, including lifting fixtures, straps,
           etc. must be certified.

          Fall protection equipment and training required for tall ladders,
           raised platforms, rooftop work, etc.
04/26/06                                                           B-Factory Operations Review
          PEP-II B-Factory operations are managed by the Accelerator Systems
           Division in the PPA Directorate.

          Control room operations are carried out by trained operators in
           compliance with documented policies and procedures for all safety-related

          Maintenance and installation work is accomplished through a matrix
           organization, drawing on engineers, technicians, and ES&H and shop
           services from the Operations Directorate.

          We have developed organizational structures, software tools, and
           documentation that enable us to manage an ambitious accelerator-based
           science program safely and effectively.

          Enhanced policies, procedures, and training programs introduced in the
           past two years ensure that all our work is carried out with high standards
           for safety.

04/26/06                                                               B-Factory Operations Review

To top