Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

IA Cooke May 2010.ppt - Wikispaces

VIEWS: 13 PAGES: 30

									   A Testbed for Studies of Team
  Cognition in the Cyber Security
              Domain
Nancy J. Cooke
Prashanth Rajivan
Shankaranarayanan Venkatanarayanan


Arizona State University
5 May 2010
                           Cooke’s Background
                 Background                                   Relevant Research
•    Education: Cognitive Psychology/Human Factors
       George Mason University, B.A.
       New Mexico State University, M.A., Ph.D.
•    Positions
       Rice University
       New Mexico State University
       Arizona State University &                    Team Cognition Military, Cyber, and
       Cognitive Engineering Research Institute
•    Applied Experience: U.S Air Force, Navy, Army,
                                                            Medical Applications
     NASA, NTSB, VA
•    Section Editor, Human Factors
•    USAF Scientific Advisory Board
•    National Research Council Committee on Human
     Systems Integration



                                                              Communication Analysis
                                                                        Sponsors
    Metrics for Coordination and Collaboration        •   Air Force Office of Scientific Research
                                                      •   Air Force Research Laboratory
                                                      •   Office of Naval Research
                                                      •   Army Research Office
                                                      •   Leonard Wood Institute
                                                      •   Veteran’s Administration – MWM VERC
              Overview
• MURI and ASU Team
• Team Cognition and Team Situation
  Awareness
• Other Team Testbeds
• CyberCog – New Testbed
    MURI: Computer-aided Human Centric
         Cyber Situation Awareness
    DoD Multidisciplinary University Research Initiative (MURI) program project,
                         funded through Army Research Office

Two fundamental limitations of Cyber Situation Awareness (C-SA)
•    Gap: human cognition < -- > C-SA tools
      – Situation data exceeds “cognitive throughput” of human analysts
• “Blind spots” in views of cyber situation for existing C-SA tools (including
  auditing, vulnerability scanners, attack graph tools, intrusion detection
  systems, damage assessment tools, and forensics tools)
Cyber-SA Vision
• Build data < -- > human decision links through innovations
      –   knowledge fusion
      –   cognitive automation
      –   artificial intelligence
      –   visual analytics
• Awareness-driven cyber defense vs. malware behavior dependent defense
• Automatic blind spot identification and monitoring techniques
                MURI Partners
•   Professor Peng Liu, Penn State University, Overall PI
•   Professor Nancy Cooke, Arizona State University
•   Professor Coty González, Carnegie Mellon University
•   Professor Dave Hall, Penn State University
•   Professor Sushil Jajodia, George Mason University
•   Professor Mike McNeese, Penn State University
•   Professor Peng Ning, NC State University
•   Professor VS Subrahmanian, Univ. of Maryland
•   Professor John Yen, Penn State University
•   Professor Michael Young, NC State University
   ASU MURI Team
Nancy J. Cooke
Professor, Cognitive Science & Engineering
College of Technology and Innovation



Prashanth Rajivan
Graduate Student
Master’s in Computing Studies
College of Technology Innovation


Shankaranarayanan Venkatanarayanan
Graduate Student
Master’s in Computing Studies
College of Technology and Innovation
Teams and Cognitive Tasks
  Team is unit of analysis = Heterogeneous
  and interdependent group of individuals
   (human or synthetic) who plan, decide,
  perceive, design, solve problems, and act
          as an integrated system.
  Cognitive activity at the team level= Team
                   Cognition
   Improved team cognition  Improved
        team/system effectiveness

  Heterogeneous = differing backgrounds,
    differing perspectives on situation
            (surgery, basketball)
              • Unmanned Aerial Vehicles
              • USS Vincennes shoots down
                Iranian airbus (1988)
   Some       • Challenger/Columbia accidents
                tied to poor organizational
Instances       decision making (1986/2003)
              • Response to 9/11 reveals
                communication breakdowns
of Failures     (2001)
              • Katrina response lacked
 of Team        coordination (2005)
              • Sago Mine disaster report cites
Cognition       poor command-and-control
                (2006)
              • VA Tech communications
                substandard (2007)
              • Friendly fire incidents
              • Various health care mishaps
                attributed to poor teamwork
 And some Miracle on the
successes… Hudson




                           Response
                           to Fargo
                           flooding
 Interactive Team Cognition in a
             Nutshell
                     Team interactions often in the
                     form of explicit
                     communications are the
                     foundation of team cognition

                   ASSUMPTIONS
1) Team cognition is an activity; not a property or
   product
2) Team cognition is inextricably tied to context
3) Team cognition is best measured and studied
   when the team is the unit of analysis
      US 2004 Olympic Basketball Team
"We  still have a couple of days, but I
don't know where we are," replied USA
head coach Larry Brown to a question
Wednesday on where his team was in
its preparations. "We have good
moments and bad, but I've got a pretty
good understanding of who needs to
play. Now the job is to get an
understanding of how we have to play."

A team of experts does NOT make
        an expert team

Collaborative skill is not additive
US 1980 Olympic Ice Hockey Team

              Herb Brooks and 20
              young “no-names”
              won the 1980
              Olympic Gold Medal
              in Ice Hockey

              An expert team made
              up of no-names…
    Our UAV
    Testbed

UAV-STE:
Uninhabited Air
Vehicle (ground
control station)
Synthetic Task
Environment
 for research on
team cognition       In our UAV STE three operators must
(DURIP 1997; USAF    coordinate over headsets in order to
funded)             maneuver their UAV to take pictures of
                               ground targets
                                            DEMPC
                Payload Operator
                                            navigator, mission
Three team      controls camera
                settings, takes photos,
                                            planner, plans
                                            route from target
members         and monitors camera
                                            to target under
                systems
with inter-                                 constraints

 dependent
tasks
                 Air Vehicle Operator
                 controls UAV airspeed,
                 heading, and altitude
                 and monitors air vehicle
                 systems

      Interdependence requires interaction,
         communication, & coordination
   Our MacroCog (Macro-Cognition
             Testbed)
 MacroCog
  Testbed
Navy-funded
lab for strategic
planning and
decision-
making in the
context of
noncombatant
evacuation
operations
MacroCog Roles in Current Experiment

  Information Personnel    Equipment
    Warfare   Specialist:  Specialist:
   Specialist  Military Land/Sea Vehicles



                                            Experimenter 1




                                            Experimenter 2




   Personnel          Equipment
   Specialist:       Specialist: Air
  Humanitarian         Vehicles
                Example of Empirical Results on
                       Team Cognition
  As teams acquire experience, performance improves, interactions improve, but not individual or
                                      collective knowledge

                                    600
                                                                                        Tm 1
                                    500                                                 Tm 2
                                                                                        Tm 3
                 Team Performance




                                                                                        Tm 4
                                    400
                                                                                        Tm 5
                                                                                        Tm 6
                                    300
                                                                                        Tm 7
                                                                                        Tm 8
                                    200                                                 Tm 9
                                                                                        Tm 10
                                    100                                                 Tm 11


                                     0
40-min missions                           1   2   3   4   5     6   7   8   9    10
                                                          Mission               Spring Break
• Individuals are trained to criterion prior to M1
• Asymptotic team performance after 4 40-min missions (robust finding)
• Knowledge changes tend to occur in early learning (M1) and stabilize
• Process improves and communication becomes more standard over time
            Team Situation Awareness
A team’s coordinated perception and action in response to a
                 change in the environment


       How can we exercise team SA in a testbed?
                 How can we measure it?
           How can we intervene to improve it?
                                          Contrary to view
                                            that all team
                                           members need
                                           to “be on the
                                             same page”
What is Meant by Coordinated
  Perception and Action?
  Measure of Team Situation Awareness
• Change is introduced (communication breakdown, enemy in area,
  storm) that will impact mission

• 2-3 team members are presented cues regarding change

• Team members need to perceive cues in a coordinated way (i.e.,
  connect the dots) to identify the change

• Team members coordinate to take action relevant to the change
  (e.g., change altitude, communicate indirectly)

• Measure in terms of outcome and process – who on team was
  involved?
      CyberCog Simulator
Web based Simulator application for
measuring individual interaction and
team collaboration (e.g., team situation
awareness) in a Cyber security analysis
situation
CyberCogSimulator – System Overview
CyberCogSimulator – Components
• Cyber Security Analyst (User)
  – Assigned a specific role such as Denial of Service
    (Dos) specialist, Malware specialist and Phishing
    specialist
  – Understands the scenario given, use events and
    attack symptoms, collaborates with other
    participants to identify a potential attack or a
    combination of attacks
  – The team reaches a common consensus on the
    type of attack and its corresponding events
CyberCogSimulator – Components
• Master controller and Evaluator
  – Queries attack scenarios, events and symptoms
    from the database
  – Distributes the events and symptoms to the
    participants
  – Logs the interaction between participants at real
    time
  – Evaluates and scores the participants findings
    with the expected results
 CyberCogSimulator – Components
• Database server
  – MySQL database server stores :-
    • Attack Scenarios
    • Events corresponding to attack scenarios
      including some false positives & noise events
    • Attack Symptoms for each specialization (E.g.,
      Dos, Malware , Phishing) identified
    • The expected results, interaction (between
      participants ) logs and attack conclusion arrived
      at by each team for each session
User and Team Views
User Screen       Common Screen

      Events         Suspicious Events

    Symptoms               Submit

       Match

      Broadcast                          Legends


       Publish                       Functions


      Unknown                            Data
CyberCog Simulator- Interaction
     CyberCogSimulator- Architecture

                                                                            Web
         Dos Specialist                                                   Services

                                            Microsoft       POCO’s
                                               IIS
                           Intra/Internet
Malware                                                              ADO.net
Specialist


                                            Controller &   Model Tier
              Phishing                       View Tier
              Specialist

             Client Tier

                                                                                 Database
                 Conclusion
• There are current gaps and limitations in Cyber
  Situation Awareness
• Cyber situation awareness by teams involves the
  coordinated perception and action in the face of
  a change in the cyber situation
• CyberCog will allow the MURI team and others
  to better understand team-based cyber SA and
  to test algorithms and tools developed for
  improving it
Team Cognition Research Program

                              UAS Field Data

                                                                                         2990



                                                                                         2980




                                                               Cumulative Speaking (s)
 Testbeds:
                                               Empirical                                 2970




 1) UAS C2                                                                               2960




                                               Studies in                                2950



 2) Navy Strategic Planning
                                               Testbed
                                                                                         2940


                                                                                                3540   3560   3580   3600    3620 3640   3660   3680   3700
                                                                                                                            Time (s)




                                                                                                              Measures




ACT-R
Model of                                          Theory Development
Synthetic Dynamical Systems Modeling
Teammate

								
To top