Sample Evaluation Forms for Collectors

Document Sample
Sample Evaluation Forms for Collectors Powered By Docstoc
					     EVALUATION PLAN

MASSACHUSETTS STATEWIDE EXERCISE
   PROGRAM – PHASE IV HAZMAT




  DISTRICT #5 TABLETOP EXERCISE
             XXXX, MA
          XXXX XXX, 2005
THIS PAGE INTENTIONALLY LEFT BLANK
                                         PREFACE
The purpose of an Evaluation Plan (EVALPLAN) is to establish a formal methodology for
evaluating an exercise. It provides guidance on how to evaluate the exercise, areas to evaluate,
criteria to be used, and the evaluation reporting process that will be used during the exercise to
determine the effectiveness of applicable plans and procedures and whether pre-identified goals
and objectives are met.

This EVALPLAN was produced with the assistance of the Exercise Design Team (EDT). This
plan is tangible evidence of the public safety partnership between the State of Massachusetts, the
Federal government, and other public and private entities to prepare for the threat of terrorist acts
using hazardous materials (Hazmat). Exercises serve as the culmination of training towards a
collective level of preparedness among all the exercise Participants. Documents such as this
EVALPLAN map the progress of improving collective readiness and response capabilities.




                                                 iii
THIS PAGE INTENTIONALLY LEFT BLANK




                iv
     ADMINISTRATIVE HANDLING INSTRUCTIONS
1.   The formal title of this document is the “Massachusetts Hazmat Tabletop Exercise
     Evaluation Plan (EVALPLAN).”

2.   This document should be handled, transmitted, and stored in accordance with appropriate
     State of Massachusetts document safeguards and only should be released to individuals
     on a need-to-know basis.

3.   Reproduction of this document in whole or in part is prohibited without the prior
     approval of the State of Massachusetts.

4.   The Massachusetts Executive Office of Public Safety Point of Contact is:

            XXXXX
            XXXXX

5.   The Massachusetts Department of Fire Services POC is:

            XXXXX
            XXXXX

6.   The Massachusetts Department of Public Health POC is:

            XXXXX
            XXXXX




                                             v
THIS PAGE INTENTIONALLY LEFT BLANK




                vi
                                             TABLE OF CONTENTS
Preface............................................................................................................................................ iii

Administrative Handling Instructions............................................................................................. v

Table of Contents.......................................................................................................................... vii

Evaluator Guidance......................................................................................................................... 9

   General Information.................................................................................................................... 9

   Purpose........................................................................................................................................ 9

   Scope........................................................................................................................................... 9

   Design Objectives ..................................................................................................................... 10

   Exercise Structure ..................................................................................................................... 10

   Player Attendees ....................................................................................................................... 11

   Roles and Responsibilities ........................................................................................................ 11

   Assumptions and Artificialities ................................................................................................ 11

   Exercise Rules........................................................................................................................... 12

   Concept of Evaluation .............................................................................................................. 12

   Evaluator Instructions ............................................................................................................... 12

   Exercise Schedule ..................................................................................................................... 14

Tabletop Exercise Evaluation Guides........................................................................................... 15

After Action Report Outline ......................................................................................................... 25

Sample Write Up Form................................................................................................................. 26

Participant Feedback..................................................................................................................... 27




                                                                         vii
THIS PAGE INTENTIONALLY LEFT BLANK




               viii
                          EVALUATOR GUIDANCE
General Information

The Massachusetts Phase IV Hazmat Tabletop Exercise (TTX) will be conducted on June XX,
2005 at the XXXX XXXX in XXX, MA. Registration for the TTX will take place from 8:30 am
to 9:00 am. The TTX will be conducted from 9:00 am to 2:00 or 2:30 pm.

The Massachusetts Department of Public Health (DPH) and the Executive Office of Public
Safety (EOPS) are sponsoring this TTX.

Purpose

The TTX will provide Players with an opportunity to evaluate Hazmat emergency response and
incident management concepts, plans, and capabilities for responding to a terrorist event in the
Hazmat Districts. Player feedback will be used to improve coordination and response functions.


Scope

The Players will participate in a TTX. Facilitated discussions will be utilized to address the
challenges presented during a hazmat terrorist incident. Discussions will focus on staff roles and
responsibilities as set forth in various Emergency Plans including the Department of Fire
Services Hazmat Response Field Operations Guide.




                                                9
Design Objectives

The Exercise Design Team (EDT) established the following exercise objectives for this TTX:
   1. Evaluate the use of the new Hazardous Materials Response Field Operations Guide by
      District Hazmat Teams.
         a. Focus on the use of the expanded command and staffing under the Incident
              Command System.
         b. Elements of command and staffing include Hazmat operations, Hazmat liaison,
              Hazmat logistics, etc.
   2. Test the working knowledge of the new rapid rescue entry procedures.
   3. Test and validate initial notification procedures.
   4. Test and validate the coordination of decontamination units, MEMA, Law Enforcement,
      Public Health, etc., with Hazmat teams.
              a)     Examine Incident Commander’s concerns with sufficient resources and
                     logistics.
              b)     Examine Law Enforcement coordination with evidence collection
                     procedures.
              c)     Examine decontamination teams to assure hospitals were protected from
                     contamination.
              d)     Examine Incident Command and Unified Command.


Exercise Structure

Exercise play will take place for approximately 3 to 4 hours and will include a background
briefing. A lunch break and “hotwash” are scheduled for during and immediately after the
exercise, respectively.
The TTX is conducted in two modules: (1) Initial Notification and Rapid Rescue Entry, (2)
Incident Coordination and Hazmat Response Operations.
Immediately following the conclusion of exercise play, the facilitator will lead Players in a
hotwash to discuss the TTX.




                                                10
Player Attendees

The following individuals will be participating in the TTX:

10 Hazmat Team Members (e.g. Leader, Safety Officer, Liaison, Planning Officer, Operations
Officer, Logistics Officer)
XXXXXX Fire (Incident Commander, Operations, Decon, Planning & Logistics)
XXXXXX Police Representative
State CST Team Representative
Mass. State Police
XXXXXX Emergency Medical Services Representative
Massachusetts Emergency Management Agency



Roles and Responsibilities

Players respond to scenario events based on expert knowledge of their agency or organization’s
response assets and current plans and procedures.

Facilitators provide situation updates and moderate discussions to the functional groups. They
also provide additional information, resolve questions as required, and make sure exercise play is
proceeding in the proper direction. Key members of the exercise design team will also assist
with facilitation as subject matter experts (SMEs) if necessary.
Evaluators gather data to evaluate how exercise play meets the TTX objectives. Evaluators will
be the primary authors of the exercise After Action Report (AAR).

Data Collectors record the discussions, decisions, and actions of each table in order to provide
documentation for post-exercise analysis and the After Action Report.

Observers only observe the exercise as they develop their responses during the caucus sessions.
Observers do not participate in the moderated discussion period.


Assumptions and Artificialities

The following assumptions and artificialities apply to this TTX. These parameters are necessary
to facilitate the accomplishment of exercise objectives within the constraints of the exercise
structure. All exercise participants will adhere to these parameters.
     •   The scenario is plausible, and events occur as they are presented.
     •   There are no “hidden agendas” or trick questions.
     •   Agencies and organizations not represented are assumed to be cooperating and
         supporting the response using their standard procedures and policies.



                                               11
Exercise Rules

This TTX is being conducted in a no-fault environment. There is no single “right” response to
scenario events and open discussion is encouraged. The exercise is an opportunity to discuss and
present multiple options and possible solutions. Participants will adhere to the following rules of
play:
     •   Participants will act in a professional manner at all times.
     •   Players will not have prior knowledge of the scenario.
     •   Players will respond as if exercise events are real, using existing plans, procedures,
         equipment, and other response assets.
     •   Response groups may discuss response actions with other response groups.

Decisions are not precedent setting and may not reflect an organization’s final position on a
given issue.



Concept of Evaluation

This exercise will be evaluated based on formal HSEEP guidelines. The purpose of the
evaluation is to determine the capability to implement certain critical HSEEP outcomes (e.g.
Emergency Assessment, Incident Site Hazard Management) in the event of a terrorist event that
involves hazardous materials. These outcomes and the specific applicable tasks under these
outcomes are contained in the section of this document entitled “Exercise Evaluation Guides.”
Since this is a discussion-based TTX, actual performance will not be observed. Instead,
evaluators will focus on documenting key elements of exercise discussions that relate to the
capability to perform those outcomes and tasks. The documented discussions will be used to
generate exercise “lessons learned” and recommendations.


Evaluator Instructions


Before the exercise:
       • Become familiar with the exercise objectives, scenario, and evaluation guides. Note
          for the detailed evaluation guides.
       • Determine which guidelines apply to different members of the exercise participants
          (e.g. identify agent task is responsibility of Hazmat personnel)

During the exercise:
       • Be unobtrusive (i.e., transparent to the Players) but ask clarifying questions
       • Observe and listen to all Player activities
       • Record key discussions related to the capability to perform the specified outcomes and
          critical tasks. The evaluator will keep records of who, what occurred, when, why, and
          how actions were conducted related to the following:



                                                  12
          o   Plans, policies and procedures
          o   Roles and responsibilities
          o   Authority to make decisions
          o   Information sharing
          o   Federal, state, and local agency roles
          o   Mutual aid agreements

After the exercise:
        • Summarize your observations in writing using the provided issue write-up forms,
           paying particular attention to the mission outcomes and critical tasks.
            o Identify strengths
            o Identify shortfalls or areas for improvement
            o Prepare recommendations for follow-up actions




                                               13
Exercise Schedule

     Pre-Exercise Schedule – XXXX X, 2005, 2pm-4pm (for facilitators/evaluators)

     Topics include:

            •   Facility Setup
            •   Exercise Overview Briefing
            •   Facilitator/Evaluator Briefing
            •   Walk through Exercise Scenario
            •   Guidance for Hot Wash and AAR Preparation
            •   Comments and Wrap-up

     Exercise Schedule – XXX XX, 2005

            8:30 – 9:00 am              Registration

            9:00 – 9:20 am              Introduction

            9:20 – 9:40 am              Purpose and Scenario Background

            9:40 – 10:40 am             Module One: Initial Notification and Rapid
                                        Rescue Entry

            10:40 – 11:00 am            Break

            11:00 – 12:30 pm            Module Two: Incident Coordination and
                                        Hazmat Response Operations

            12:30 – 1:15 pm             Lunch

            1:15 – 2:00 or 2:30 pm      Hot wash for Participants and Closing Comments




                                          14
TABLETOP EXERCISE EVALUATION GUIDES




                15
16
17
18
19
20
21
22
23
24
                AFTER ACTION REPORT OUTLINE

   I. Executive Summary

  II. Exercise Overview

 III. Exercise Goals & Objectives

 IV. Exercise Events Synopsis

  V. Analysis of Mission Outcomes

 VI. Analysis of Critical Task Performance (Issue write-ups)

VII. Conclusion

VIII. Appendix A: Improvement Plan Matrix (compiled from Issue write-ups)

 IX. Appendix B: Exercise Design Critique




                                            25
                      SAMPLE WRITE UP FORM
Task: (number and title of EEG)

Issue #__: Title: (A consecutive number, to be assigned later by Evaluation Team
Leader, and a one-sentence title.)


Reference: (Citation to source of the issue in local plan, procedure, MOU, EEG, etc.)



Summary: (1 to 2 sentences or a brief paragraph description of the issue.)




Consequence: (The effects of the issue on the ability of the community to respond, as
generally described in the appropriate EEG.)



Analysis: (What happened? What was supposed to happen? Why was there a
difference (the basic underlying causal factor; see Root Cause Analysis)? What should
be learned?)




Recommendations: (What can be done to resolve the issue, including changes in
plans or procedures, organizational structures, leadership and management processes,
training, additional equipment or resources, etc.)




                                          26
PARTICIPANT FEEDBACK




         27

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:40
posted:8/2/2011
language:English
pages:27
Description: Sample Evaluation Forms for Collectors document sample