Learning Center
Plans & pricing Sign in
Sign Out

IA Cooke May 2010.ppt - Wikispaces


									   A Testbed for Studies of Team
  Cognition in the Cyber Security
Nancy J. Cooke
Prashanth Rajivan
Shankaranarayanan Venkatanarayanan

Arizona State University
5 May 2010
                           Cooke’s Background
                 Background                                   Relevant Research
•    Education: Cognitive Psychology/Human Factors
       George Mason University, B.A.
       New Mexico State University, M.A., Ph.D.
•    Positions
       Rice University
       New Mexico State University
       Arizona State University &                    Team Cognition Military, Cyber, and
       Cognitive Engineering Research Institute
•    Applied Experience: U.S Air Force, Navy, Army,
                                                            Medical Applications
•    Section Editor, Human Factors
•    USAF Scientific Advisory Board
•    National Research Council Committee on Human
     Systems Integration

                                                              Communication Analysis
    Metrics for Coordination and Collaboration        •   Air Force Office of Scientific Research
                                                      •   Air Force Research Laboratory
                                                      •   Office of Naval Research
                                                      •   Army Research Office
                                                      •   Leonard Wood Institute
                                                      •   Veteran’s Administration – MWM VERC
• MURI and ASU Team
• Team Cognition and Team Situation
• Other Team Testbeds
• CyberCog – New Testbed
    MURI: Computer-aided Human Centric
         Cyber Situation Awareness
    DoD Multidisciplinary University Research Initiative (MURI) program project,
                         funded through Army Research Office

Two fundamental limitations of Cyber Situation Awareness (C-SA)
•    Gap: human cognition < -- > C-SA tools
      – Situation data exceeds “cognitive throughput” of human analysts
• “Blind spots” in views of cyber situation for existing C-SA tools (including
  auditing, vulnerability scanners, attack graph tools, intrusion detection
  systems, damage assessment tools, and forensics tools)
Cyber-SA Vision
• Build data < -- > human decision links through innovations
      –   knowledge fusion
      –   cognitive automation
      –   artificial intelligence
      –   visual analytics
• Awareness-driven cyber defense vs. malware behavior dependent defense
• Automatic blind spot identification and monitoring techniques
                MURI Partners
•   Professor Peng Liu, Penn State University, Overall PI
•   Professor Nancy Cooke, Arizona State University
•   Professor Coty González, Carnegie Mellon University
•   Professor Dave Hall, Penn State University
•   Professor Sushil Jajodia, George Mason University
•   Professor Mike McNeese, Penn State University
•   Professor Peng Ning, NC State University
•   Professor VS Subrahmanian, Univ. of Maryland
•   Professor John Yen, Penn State University
•   Professor Michael Young, NC State University
   ASU MURI Team
Nancy J. Cooke
Professor, Cognitive Science & Engineering
College of Technology and Innovation

Prashanth Rajivan
Graduate Student
Master’s in Computing Studies
College of Technology Innovation

Shankaranarayanan Venkatanarayanan
Graduate Student
Master’s in Computing Studies
College of Technology and Innovation
Teams and Cognitive Tasks
  Team is unit of analysis = Heterogeneous
  and interdependent group of individuals
   (human or synthetic) who plan, decide,
  perceive, design, solve problems, and act
          as an integrated system.
  Cognitive activity at the team level= Team
   Improved team cognition  Improved
        team/system effectiveness

  Heterogeneous = differing backgrounds,
    differing perspectives on situation
            (surgery, basketball)
              • Unmanned Aerial Vehicles
              • USS Vincennes shoots down
                Iranian airbus (1988)
   Some       • Challenger/Columbia accidents
                tied to poor organizational
Instances       decision making (1986/2003)
              • Response to 9/11 reveals
                communication breakdowns
of Failures     (2001)
              • Katrina response lacked
 of Team        coordination (2005)
              • Sago Mine disaster report cites
Cognition       poor command-and-control
              • VA Tech communications
                substandard (2007)
              • Friendly fire incidents
              • Various health care mishaps
                attributed to poor teamwork
 And some Miracle on the
successes… Hudson

                           to Fargo
 Interactive Team Cognition in a
                     Team interactions often in the
                     form of explicit
                     communications are the
                     foundation of team cognition

1) Team cognition is an activity; not a property or
2) Team cognition is inextricably tied to context
3) Team cognition is best measured and studied
   when the team is the unit of analysis
      US 2004 Olympic Basketball Team
"We  still have a couple of days, but I
don't know where we are," replied USA
head coach Larry Brown to a question
Wednesday on where his team was in
its preparations. "We have good
moments and bad, but I've got a pretty
good understanding of who needs to
play. Now the job is to get an
understanding of how we have to play."

A team of experts does NOT make
        an expert team

Collaborative skill is not additive
US 1980 Olympic Ice Hockey Team

              Herb Brooks and 20
              young “no-names”
              won the 1980
              Olympic Gold Medal
              in Ice Hockey

              An expert team made
              up of no-names…
    Our UAV

Uninhabited Air
Vehicle (ground
control station)
Synthetic Task
 for research on
team cognition       In our UAV STE three operators must
(DURIP 1997; USAF    coordinate over headsets in order to
funded)             maneuver their UAV to take pictures of
                               ground targets
                Payload Operator
                                            navigator, mission
Three team      controls camera
                settings, takes photos,
                                            planner, plans
                                            route from target
members         and monitors camera
                                            to target under
with inter-                                 constraints

                 Air Vehicle Operator
                 controls UAV airspeed,
                 heading, and altitude
                 and monitors air vehicle

      Interdependence requires interaction,
         communication, & coordination
   Our MacroCog (Macro-Cognition
lab for strategic
planning and
making in the
context of
MacroCog Roles in Current Experiment

  Information Personnel    Equipment
    Warfare   Specialist:  Specialist:
   Specialist  Military Land/Sea Vehicles

                                            Experimenter 1

                                            Experimenter 2

   Personnel          Equipment
   Specialist:       Specialist: Air
  Humanitarian         Vehicles
                Example of Empirical Results on
                       Team Cognition
  As teams acquire experience, performance improves, interactions improve, but not individual or
                                      collective knowledge

                                                                                        Tm 1
                                    500                                                 Tm 2
                                                                                        Tm 3
                 Team Performance

                                                                                        Tm 4
                                                                                        Tm 5
                                                                                        Tm 6
                                                                                        Tm 7
                                                                                        Tm 8
                                    200                                                 Tm 9
                                                                                        Tm 10
                                    100                                                 Tm 11

40-min missions                           1   2   3   4   5     6   7   8   9    10
                                                          Mission               Spring Break
• Individuals are trained to criterion prior to M1
• Asymptotic team performance after 4 40-min missions (robust finding)
• Knowledge changes tend to occur in early learning (M1) and stabilize
• Process improves and communication becomes more standard over time
            Team Situation Awareness
A team’s coordinated perception and action in response to a
                 change in the environment

       How can we exercise team SA in a testbed?
                 How can we measure it?
           How can we intervene to improve it?
                                          Contrary to view
                                            that all team
                                           members need
                                           to “be on the
                                             same page”
What is Meant by Coordinated
  Perception and Action?
  Measure of Team Situation Awareness
• Change is introduced (communication breakdown, enemy in area,
  storm) that will impact mission

• 2-3 team members are presented cues regarding change

• Team members need to perceive cues in a coordinated way (i.e.,
  connect the dots) to identify the change

• Team members coordinate to take action relevant to the change
  (e.g., change altitude, communicate indirectly)

• Measure in terms of outcome and process – who on team was
      CyberCog Simulator
Web based Simulator application for
measuring individual interaction and
team collaboration (e.g., team situation
awareness) in a Cyber security analysis
CyberCogSimulator – System Overview
CyberCogSimulator – Components
• Cyber Security Analyst (User)
  – Assigned a specific role such as Denial of Service
    (Dos) specialist, Malware specialist and Phishing
  – Understands the scenario given, use events and
    attack symptoms, collaborates with other
    participants to identify a potential attack or a
    combination of attacks
  – The team reaches a common consensus on the
    type of attack and its corresponding events
CyberCogSimulator – Components
• Master controller and Evaluator
  – Queries attack scenarios, events and symptoms
    from the database
  – Distributes the events and symptoms to the
  – Logs the interaction between participants at real
  – Evaluates and scores the participants findings
    with the expected results
 CyberCogSimulator – Components
• Database server
  – MySQL database server stores :-
    • Attack Scenarios
    • Events corresponding to attack scenarios
      including some false positives & noise events
    • Attack Symptoms for each specialization (E.g.,
      Dos, Malware , Phishing) identified
    • The expected results, interaction (between
      participants ) logs and attack conclusion arrived
      at by each team for each session
User and Team Views
User Screen       Common Screen

      Events         Suspicious Events

    Symptoms               Submit


      Broadcast                          Legends

       Publish                       Functions

      Unknown                            Data
CyberCog Simulator- Interaction
     CyberCogSimulator- Architecture

         Dos Specialist                                                   Services

                                            Microsoft       POCO’s

                                            Controller &   Model Tier
              Phishing                       View Tier

             Client Tier

• There are current gaps and limitations in Cyber
  Situation Awareness
• Cyber situation awareness by teams involves the
  coordinated perception and action in the face of
  a change in the cyber situation
• CyberCog will allow the MURI team and others
  to better understand team-based cyber SA and
  to test algorithms and tools developed for
  improving it
Team Cognition Research Program

                              UAS Field Data



                                                               Cumulative Speaking (s)
                                               Empirical                                 2970

 1) UAS C2                                                                               2960

                                               Studies in                                2950

 2) Navy Strategic Planning

                                                                                                3540   3560   3580   3600    3620 3640   3660   3680   3700
                                                                                                                            Time (s)


Model of                                          Theory Development
Synthetic Dynamical Systems Modeling

To top