Docstoc

Alan_Spiker

Document Sample
Alan_Spiker Powered By Docstoc
					                                Scenario-Based Training:
                     Lessons from Development and Implementation


                                                              7 November 2006


                                        Presentation to:
                      Storytelling as an Instructional Method Conference
                                         Mesa, Arizona


                                                            Presentation by:
                                                              Alan Spiker
                                                          Anacapa Sciences, Inc.
                                                           Santa Barbara, CA

                                                                                   1
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                          Topics

     What’s a Scenario and why is it needed?

     Challenges Facing a Small Business Implementing SBT

     Evolution of Scenarios as an Element of Training

     Useful ISD Principles of Scenario Based Training

     Measuring SBT Effectiveness

     Lessons Learned: Navy, Commercial Aviation, Army

                                                                   2
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                          Topics (details)

           What’s a Scenario and why is it needed for training?
                  - look to the Thesaurus
                  - many good reasons and some not-so good reasons for scenarios
           Challenges Facing a Small Business in Implementing SBT
                  - cajole not coerce, evince not convince, insinuate not dictate
                  - implementing various types of SBIR-funded technologies
           Evolution of Scenarios as an Element of Training
                  - Geospecific DB --- > Framework ---- > Wizard ----- > Serious Games?
           Useful ISD Principles of Scenario Based Training
                  - eight that REALLY matter
           Measuring SBT Effectiveness
                  - Kirkpatrick’s 4-Level formal method
                  - informal method in the real world
           Lessons from Implementing SBIRs w/ Navy, Commercial Aviation, Army
                  - Sometimes a great notion
                  - Our scorecard: wins, losses, draws

                                                                                          3
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                What’s a Scenario?
         1.            An outline of the plot of a dramatic work, giving
                       particulars to scenes, characters, situations, etc.
         2.            The outline or manuscript of a motion picture or television
                       program, giving the action in the order in which it takes
                       place, description of scenes and characters
         3.            An imagined or projected sequence of events, esp. any of
                       several detailed plans or possibilities
         4.            Setting for a work of art or literature
         5.            Postulated sequence of possible events
         6.            An outline or synopsis of a play
                       Syn: scheme, plan, concept, sketch
           Sources: Dictionary.com WordNet
                                                                                     4
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                  What’s a Scenario really?
         1.            An outline of the plot of a dramatic work, giving
                       particulars to scenes, characters, situations, etc.
         2.            The outline or manuscript of a motion picture or television
                       program, giving the action in the order in which it takes
                       place, description of scenes and characters
         3.            An imagined or projected sequence of events, esp. any of
                       several detailed plans or possibilities
         4.            Setting for a work of art or literature
         5.            Postulated sequence of possible events
         6.            An outline or synopsis of a play
         7.            Syn: scheme, plan, concept, sketch
           Sources: Dictionary.com WordNet
                                                                                     5
Storytelling as Instruction   Nov 06   Anacapa Sciences
                       Scenarios are good because they:


         •             Can be a script or a blueprint of what will happen
         •             Define roles of non-trainees
         •             Constrain allowable actions, decision points
         •             Help establish cognitive realism and trainee buy-in
         •             Lay out sequence of action to unfold
         •             They define which events occur, and when
         •             Give the storyline of actions
         •             Serve as communication conduit for training team
         •             And thus ……

                                                                             6
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                           Scenarios:
        •            Prevent chaos in the training environment
        •            Provide an antidote to unscripted free play in the sims
        •            Make training more interesting to the trainee
        •            Make training more believable – more face valid
        •            Make training more personal, customizable
        •            Are more consistent with our Experiential Learning Model
                     approach to training adults

                                                          BUT Scenarios --
        •            Can’t substitute for measurable learning objectives
        •            Don’t ensure criterion-based training
        •            Don’t guarantee positive transfer of training
                                                                                7
Storytelling as Instruction   Nov 06   Anacapa Sciences
                  Our Challenge as a Small Business

    • Anacapa Sciences – small business est. 1969
    • Implement SBT as part of SBIR products
    • Solicit claimants through insinuation, use of data-
      driven approach, value-added consulting on the side
      - so no organizational clout but that’s hardly unique to us
    • Since 2000, attempted SBT in several SBIRs:
              –     SamePage: team training in Shared Understanding for ARI (Ph. II)
              –     SimDATT: scenario-generation tool for NavAir (Ph. II+)
              –     (CT)2: web-based critical thinking training modules for ARI (Ph. III)
              –     TARGET: serious game based trainer (GBT) for ONR (Ph. I)
              –     Stalwart: Wiki-based team trainer for JFCOM (Ph. I)
                                                                                            8
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                 Common Approach

             SBT is still training - Typical Steps in SBT Projects
•      Identifying training need – why SBT needed in first place
•      Finding suitable SBT sponsor – how SBT will help him/her
•      Defining training baseline – what SBT is replacing
•      Finding training objectives – usually a real scavenger hunt
•      Selecting form of SBT – one size does not fit all
•      Managing expectations – what SBT can and can’t do
•      Measuring SBT effectiveness – how to know if it’s working
•      Implementing SBT – what’s the exit strategy
•      Assessing feasibility– who’s planning the birthday party
                                                                     9
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                Evolution of SBT
                                        Small Business Perspective

   • Scenarios WERE: Big, physical, fixed, geo-specific,
     1-1 physical fidelity with criterion environment,
     created by SMEs, little measurement
   • Scenarios ARE NOW: Smaller, cognitive, dynamic,
     regional, 1-1 cognitive fidelity with criterion
     environment, created by project team, stopping
     points for measurement
   • Scenarios WILL BECOME: Micro, meta-cognitive,
     adaptive, generic, 1-1 higher order fidelity with
     criterion environment, created by anyone,
     embedded measurement
                                                                     10
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                Evolution of SBT
                                                          Mission Scenario




             trainee’s role




                                                                             11
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                Evolution of SBT
                                                          Instructor Script




                                                                              instructor’s
                                                                              actions




                                                                                             12
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                           Evolution of SBT
                                Characteristic Features Now



             • Synopsis – Storyline method of
               organization
             • Trainee role specified through multiple
               means
             • Timeline – used as scaffold on which to
               specify sequence of action
             • Events – are specified, selected, and inserted
                    as basic unit of action
                                                                13
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                           Evolution of SBT
                                           Synopses - Storylines


                                                                   Select scenario from
                                                                   a library of possible
                                                                   synopses




                                                                                           14
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                            Evolution of SBT
                                                          Trainee Roles

            Conveyed through instructions, tables, special OPORDs,
            maps, immersion exercises, vignettes
 (1)        Evaluation of possible enemy courses of action that might disrupt our purpose in
            occupying Fwd 1. The enemy’s primary methods of disruption are to promote fear and
            distrust – fear of them and distrust of US forces. Determine whether US occupation of
            Fwd 1 will diminish, enhance, or remain unaffected the enemy’s capacity to:
                 Conduct criminal actions against the general populace to induce fear in them.
                 Conduct criminal actions against local leadership to induce fear in them.
                 Conduct criminal actions against the general populace and local leadership to
                     demonstrate lack of capacity of US forces to protect them.
                 Spread rumors that the US forces lack resolve to occupy Fwd 1 over a long period
                     of time.

    Responsibilities on team

             Estimate proactive and reactive enemy activities
             Predict the general enemy reaction to the positioning of our forces in Fwd 1.
             Build and interpret a link diagram that will describe the relations among and
              between civil and religious leaders.
             Estimate the threat to logistics routes when asked by the Logistics Analyst.

    Information/knowledge available to you in this mission

    1. The Bn S-2 and the commander have pondered the nature of the enemy over the past
       several months. What is listed on the OPORD, paragraph 1a is only a summary of
       their thoughts. Here is a more detailed general estimate:


                                                                                                     15
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                           Evolution of SBT
                                        Timelines – Action Flow




                                                                  16
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                           Evolution of SBT
                        Event Selection and Specification




                                                              17
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                         ISD Principles of SBT
                                                                  Overview

     • Navair funded add-on to SimDATT project
     • Identified 69 principles from ISD realm that could be applied
       to guide SBT development
     • User version incorporated into SimDATT to aid designer
                                         Design Principle 1-4: Promote student motivation by instilling mental conflict.

                                         Mental conflict can be provoked in students through several methods, including:
                                         1) using facts that contradict their past experience, 2) involving a sense of
                                         mystery by presenting unresolved problems, and 3) providing scenario events
                                         that “baffle” their perceptions.
                                         Explanation:          Provoking            GARDEN PATH EXPERIENCE
                                         mental conflict can stimulate               THAT PROMOTES MENTAL
                                                                                           CONFLICT
                                                                                                                    TYPE OF KSA

                                         a student's curiosity about                 Progressive increase in        Prelude to learning to
                                         the KSA he/she is about to                  task difficulty until STUD’s
                                                                                     skill level is exceeded
                                                                                                                    make precision
                                                                                                                    approaches
                                         learn    and     hence      their
                                         motivation to learn. These                  Presentation of ambiguous
                                                                                     stimulus that defies
                                                                                                                    Prelude to learning VFR to
                                         “garden path” experiences,                  accurate ID
                                                                                                                    IFR transition


                                         where        students        are            Presentation of conflicting    Prelude to learning

                                         essentially lured into an initial           information that makes
                                                                                     unambiguous decision
                                                                                                                    advanced aviation
                                                                                                                    decision making skills
                                         failure experience, help to                 difficult                      (e.g., divert airfield)

                                         form more accurate mental                   Presentation of benign         Prelude to learning to
                                                                                     stimulus that is followed by   recover from unusual
                                         models     and     help    them             abrupt malevolent change       attitude

                                         unlearn previously erroneous
                                         methods of looking at situa-
                                                                                     Presentation of message
                                                                                     that contains clauses which    Prelude to learning ATC
                                         tions and problems.                         are logically or factually
                                                                                     inconsistent
                                                                                                                    communication protocols


                                         Example:        There    are   a
                                         number of effective ways to
                                         trick a student to follow down
                                         a “garden path” in order to get their attention and impress upon them the
                                         difficulty of the to-be-learned KSA. Examples of five effective “garden path” tech-
                                         niques are shown in Figure 1-4.
                                         Reference: Keller & Burkman (1993); Ross, Phillips, Klein, & Cohn (2005)
                                                                                                                                                 18
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                         ISD Principles of SBT
                                          Eight Really Useful Ones


                                                          ISD Design Principle
                   1-1                 Use 5-part scenario structure
                   1-5                 Link scenario objectives to student learning goals
                   2-1                 Link scenario content to training objectives
                   2-8                 Include meta-cognitive skills in scenario KSAs
                   3-2                 Organize training by varying setup conditions
                   4-2                 Use ‘telescoping’ approach to design scenario content
                   6-7                 Use analogous cases as training scaffold
                   8-4                 Stress critical thinking in debrief questions
                                                                                               19
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                         ISD Principles of SBT
                                   Five-Part Scenario Structure


                    Storyline/Synopsis                    Players        Events                Content              Timeline

          Basic daytime flight from Dulles to             ATC        Interrupted Start                           Start & Taxi —
          Nashville. During start, several                                                    SID Chart             Interrupted Start
          interrupted start problems are                  ATIS       IFR Standard Departure
          encountered and discussed. Normal                                                   CFM (company flight Departure–
          SID using autopilot. Practice area                         HYD 1 Low Pressure       manual)                IFR
                                                          Dispatch
          entry for steep turns and stalls. In
          practice area, low hydraulic pressure                      Steep Turns              Checklists         Enroute–
          (1) light comes on. After isolation                                                                     Turns
          and correction, descent is initiated                       Stalls                                       Stalls
          back to Dulles, using autopilot. Full                                                                   HYD
          stop landing is completed.
                                                                                                                 Approach–
                                                                                                                   ILS




                                                                                                                                        20
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                           ISD Principles of SBT
                              Link Objectives to Student Goals

                                                                 After completing the
                                  The scenario will:
                                                              scenario you will be able to

                                         Expose you to
                                                                    Know bold-faced items by
                                        different types of          heart and engage correct
                                       Hydraulic Systems            emergency systems within
                                              failures                  sec. of detection




                                         Subject you to           Initiate VFR-IFR transition
                                        different levels of          actions w/o hesitation
                                       low-vis conditions          while remaining spatially
                                                                            oriented


                                       Require you to
                                      practice multiple           Internalize decision height
                                  approaches and make                 actions and opt for
                                   landings on runways               missed approach w/o
                                     of various lengths                    hesitation



                                       Expose you to high          ―Tune‖ your attention to
                                       volume ATC traffic          particular voices on the
                                         under different         radio and to assertively ask
                                        amounts of radio          for a repeat transmission
                                             noise                   when messages are
                                                                           inaudible




                                                                                                21
Storytelling as Instruction   Nov 06    Anacapa Sciences
                                         ISD Principles of SBT
                                       Link Content to Objectives




                                                                    22
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                         ISD Principles of SBT
                                 Include Meta-Cognitive Skills


              Generate mental simulations to troubleshoot emergencies
              Generate mental simulations to what-if a plan
              Generate mental simulations to ID route weaknesses
              Engage in counterfactual thinking (assume plan is wrong)
              Learn to size up a situation
              Generate story to explain away inconvenient data
              Diagnose cause of some flight disturbance
              Controlling attention through eye-scanning techniques
              Stress resistance training
              Shared situational awareness




                                                                         23
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                         ISD Principles of SBT
                                             Vary Setup Conditions


       Variables that Promote
       Transfer of Training                                 Example of Application in SBT
      High Level of Initial Learning                      Ensure that all core tasks receive many practice trials

                                                          Ensure that students are made aware of the task elements in training that
      High Perceived Similarity between
                                                          will be present on the job (e.g., time pressure, equipment malfunctions)
      Training Task and Criterion Task
                                                          Factors like time of day, amount of rest, time to prepare, etc. should be
      Tasks Learned in Relevant Context                   similar in training to that expected on the job

      Tasks Learned with Similar Cognitive                Ensure that tasks require a comparable level of critical thinking, decision
      Processes to Criterion Task                         making, problem solving, etc. as that expected on the job

                                                          Conduct training where environmental, aircraft, and flight conditions
      Variability in Training Conditions
                                                          vary widely from session to session
      Increase Difficulty of Instruction                  Include contextual factors that make learning difficult, like frequent
      through Contextual Interference                     malfunctions, radio traffic, garbled comms, etc.
      Gradually Fade out Feedback                         After initial sessions, don’t give student feedback after every takeoff/
      and Knowledge of Results                            landing, but only after several




                                                                                                                                        24
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                 ISD Principles of SBT
                   Use Telescoping Design to Create Content
                                                           (1)




                                                                 (2)



           (3)




                                                                       (5)




                              (4)
                                                                             25
Storytelling as Instruction    Nov 06   Anacapa Sciences
                                         ISD Principles of SBT
                        Use Analogous Cases as a Scaffold



                                                                 2 scenarios most like
                                                                 your present scenario
                                                                 in their emphasis on
                                                                 CRM




                                                                                    26
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                               ISD Principles of SBT
                          Stress Critical Thinking in the Debrief




                                                                       27
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                  Measuring SBT Effectiveness
                                                            The Formal Way

                                  Kirkpatrick’s 4-Level Approach to Evaluation


                                               Evaluation Level              Methodology

                        I                           User Acceptance             Survey

                       II                                 User Learning   Comprehension Test

                      III                             Transfer to Job     Gaining Unit Survey

                     IV                       Organizational Impact       Longitudinal Study

                                                                                                28
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                   Measuring SBT Effectiveness
                                          Real World Metrics - Checklist


             Does system have its own room?
             Does system have a real-person as champion?
             Does using community pony up some of its own money to keep
               system going?
             Are they establishing training objectives for each scenario?
             Do they call us (for tech support) to keep system going?
             Do they return our phone calls and e-mails?
             Do they invite us to their parties?

                                                                            29
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                     Lessons Learned
                                                          Anacapa’s Scorecard


     • Why did SimDATT version for Navy fail?
     • Why did SimDATT version for ASU-Mesa succeed?
     • Why did SamePage come to a draw with the Army?
     • Why might (CT)2 win/lose/draw with Army?
        - too close to call now


                              Our Record is 1-1-1-1


                                                                                30
Storytelling as Instruction   Nov 06   Anacapa Sciences
                              Why SimDATT Failed with Navy

      • Users liked the aviator interface and content
      •     But didn’t need scenarios for their part-task training
      •     Either FRS or refresher
      •     So all the marks appear high in Level I-II evaluation
      •     But our real-world metrics are very poor




                                                                     31
Storytelling as Instruction   Nov 06   Anacapa Sciences
                     Why SimDATT Succeeded at ASU-Mesa

   • Same basic system as for the Navy
   •     ASU-Mesa embraced scenarios for their students
   •     Make even initial training LOFT-like
   •     Have intimate relationship with the “gaining unit”
   •     Supporting a four-level Kirkpatrick evaluation




                                                              32
Storytelling as Instruction   Nov 06   Anacapa Sciences
                          Why SamePage Stalled with Army

       • Users liked the concept
       •     Users liked the immersion experience
       •     Users liked the challenge, great for leader development
       •     But need domain changes to cover multiple MOSs
       •     Needed a wizard, not a framework




                                                                  33
Storytelling as Instruction   Nov 06   Anacapa Sciences
                              What We Needed Was Something
                                        Like this




                                                             34
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                          And This




                                                                     35
Storytelling as Instruction   Nov 06   Anacapa Sciences
                          Why (CT)2 Still in Limbo with Army

    Have conducted multiple evaluations, all very positive
    Contains multitude of cognitive vignettes
    But where to put into curriculum?
    What has to drop out to put (CT)2 in?
    Yet our inputs reached other parts of CGSC curriculum




                                                               36
Storytelling as Instruction   Nov 06   Anacapa Sciences
                                                          Trends in SBT
                                                            Wrap-up

      • SBT becoming more adaptive
      • SBT could become more meta-cognitively oriented
      • SBT will become more SCORMED
      • SBT scenarios will be updated often, perhaps daily
      • SBT will become more personalized (e.g., wikis,
        avatars)
      • SBT will become “jazzed up” to support a market-
        based approach to training (how many hits did you get
              today Mr. Training Manager?)

      • SBT will become SBD, scenario based development
                                                                          37
Storytelling as Instruction   Nov 06   Anacapa Sciences
                          Meta-Cognitive Skills in Scenarios
                                             Effects Based Thinking Tool




                                                                           38
Storytelling as Instruction   Nov 06   Anacapa Sciences
                          Meta-Cognitive Skills in Scenarios
                                                          Use of Wikis



                                                                   Key Features
                                                                   •     Search
                                                                   •     Edit
                                                                   •     Save entries
                                                                   •     Color code edit versions
                                                                   •     url is:

                                                                   http://www.anacapatechnical.com/projects/jtf




                                                                                                              39
Storytelling as Instruction   Nov 06   Anacapa Sciences

				
DOCUMENT INFO