VIEWS: 13 PAGES: 30 POSTED ON: 9/8/2011
A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State University 5 May 2010 Cooke’s Background Background Relevant Research • Education: Cognitive Psychology/Human Factors George Mason University, B.A. New Mexico State University, M.A., Ph.D. • Positions Rice University New Mexico State University Arizona State University & Team Cognition Military, Cyber, and Cognitive Engineering Research Institute • Applied Experience: U.S Air Force, Navy, Army, Medical Applications NASA, NTSB, VA • Section Editor, Human Factors • USAF Scientific Advisory Board • National Research Council Committee on Human Systems Integration Communication Analysis Sponsors Metrics for Coordination and Collaboration • Air Force Office of Scientific Research • Air Force Research Laboratory • Office of Naval Research • Army Research Office • Leonard Wood Institute • Veteran’s Administration – MWM VERC Overview • MURI and ASU Team • Team Cognition and Team Situation Awareness • Other Team Testbeds • CyberCog – New Testbed MURI: Computer-aided Human Centric Cyber Situation Awareness DoD Multidisciplinary University Research Initiative (MURI) program project, funded through Army Research Office Two fundamental limitations of Cyber Situation Awareness (C-SA) • Gap: human cognition < -- > C-SA tools – Situation data exceeds “cognitive throughput” of human analysts • “Blind spots” in views of cyber situation for existing C-SA tools (including auditing, vulnerability scanners, attack graph tools, intrusion detection systems, damage assessment tools, and forensics tools) Cyber-SA Vision • Build data < -- > human decision links through innovations – knowledge fusion – cognitive automation – artificial intelligence – visual analytics • Awareness-driven cyber defense vs. malware behavior dependent defense • Automatic blind spot identification and monitoring techniques MURI Partners • Professor Peng Liu, Penn State University, Overall PI • Professor Nancy Cooke, Arizona State University • Professor Coty González, Carnegie Mellon University • Professor Dave Hall, Penn State University • Professor Sushil Jajodia, George Mason University • Professor Mike McNeese, Penn State University • Professor Peng Ning, NC State University • Professor VS Subrahmanian, Univ. of Maryland • Professor John Yen, Penn State University • Professor Michael Young, NC State University ASU MURI Team Nancy J. Cooke Professor, Cognitive Science & Engineering College of Technology and Innovation Prashanth Rajivan Graduate Student Master’s in Computing Studies College of Technology Innovation Shankaranarayanan Venkatanarayanan Graduate Student Master’s in Computing Studies College of Technology and Innovation Teams and Cognitive Tasks Team is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide, perceive, design, solve problems, and act as an integrated system. Cognitive activity at the team level= Team Cognition Improved team cognition Improved team/system effectiveness Heterogeneous = differing backgrounds, differing perspectives on situation (surgery, basketball) • Unmanned Aerial Vehicles • USS Vincennes shoots down Iranian airbus (1988) Some • Challenger/Columbia accidents tied to poor organizational Instances decision making (1986/2003) • Response to 9/11 reveals communication breakdowns of Failures (2001) • Katrina response lacked of Team coordination (2005) • Sago Mine disaster report cites Cognition poor command-and-control (2006) • VA Tech communications substandard (2007) • Friendly fire incidents • Various health care mishaps attributed to poor teamwork And some Miracle on the successes… Hudson Response to Fargo flooding Interactive Team Cognition in a Nutshell Team interactions often in the form of explicit communications are the foundation of team cognition ASSUMPTIONS 1) Team cognition is an activity; not a property or product 2) Team cognition is inextricably tied to context 3) Team cognition is best measured and studied when the team is the unit of analysis US 2004 Olympic Basketball Team "We still have a couple of days, but I don't know where we are," replied USA head coach Larry Brown to a question Wednesday on where his team was in its preparations. "We have good moments and bad, but I've got a pretty good understanding of who needs to play. Now the job is to get an understanding of how we have to play." A team of experts does NOT make an expert team Collaborative skill is not additive US 1980 Olympic Ice Hockey Team Herb Brooks and 20 young “no-names” won the 1980 Olympic Gold Medal in Ice Hockey An expert team made up of no-names… Our UAV Testbed UAV-STE: Uninhabited Air Vehicle (ground control station) Synthetic Task Environment for research on team cognition In our UAV STE three operators must (DURIP 1997; USAF coordinate over headsets in order to funded) maneuver their UAV to take pictures of ground targets DEMPC Payload Operator navigator, mission Three team controls camera settings, takes photos, planner, plans route from target members and monitors camera to target under systems with inter- constraints dependent tasks Air Vehicle Operator controls UAV airspeed, heading, and altitude and monitors air vehicle systems Interdependence requires interaction, communication, & coordination Our MacroCog (Macro-Cognition Testbed) MacroCog Testbed Navy-funded lab for strategic planning and decision- making in the context of noncombatant evacuation operations MacroCog Roles in Current Experiment Information Personnel Equipment Warfare Specialist: Specialist: Specialist Military Land/Sea Vehicles Experimenter 1 Experimenter 2 Personnel Equipment Specialist: Specialist: Air Humanitarian Vehicles Example of Empirical Results on Team Cognition As teams acquire experience, performance improves, interactions improve, but not individual or collective knowledge 600 Tm 1 500 Tm 2 Tm 3 Team Performance Tm 4 400 Tm 5 Tm 6 300 Tm 7 Tm 8 200 Tm 9 Tm 10 100 Tm 11 0 40-min missions 1 2 3 4 5 6 7 8 9 10 Mission Spring Break • Individuals are trained to criterion prior to M1 • Asymptotic team performance after 4 40-min missions (robust finding) • Knowledge changes tend to occur in early learning (M1) and stabilize • Process improves and communication becomes more standard over time Team Situation Awareness A team’s coordinated perception and action in response to a change in the environment How can we exercise team SA in a testbed? How can we measure it? How can we intervene to improve it? Contrary to view that all team members need to “be on the same page” What is Meant by Coordinated Perception and Action? Measure of Team Situation Awareness • Change is introduced (communication breakdown, enemy in area, storm) that will impact mission • 2-3 team members are presented cues regarding change • Team members need to perceive cues in a coordinated way (i.e., connect the dots) to identify the change • Team members coordinate to take action relevant to the change (e.g., change altitude, communicate indirectly) • Measure in terms of outcome and process – who on team was involved? CyberCog Simulator Web based Simulator application for measuring individual interaction and team collaboration (e.g., team situation awareness) in a Cyber security analysis situation CyberCogSimulator – System Overview CyberCogSimulator – Components • Cyber Security Analyst (User) – Assigned a specific role such as Denial of Service (Dos) specialist, Malware specialist and Phishing specialist – Understands the scenario given, use events and attack symptoms, collaborates with other participants to identify a potential attack or a combination of attacks – The team reaches a common consensus on the type of attack and its corresponding events CyberCogSimulator – Components • Master controller and Evaluator – Queries attack scenarios, events and symptoms from the database – Distributes the events and symptoms to the participants – Logs the interaction between participants at real time – Evaluates and scores the participants findings with the expected results CyberCogSimulator – Components • Database server – MySQL database server stores :- • Attack Scenarios • Events corresponding to attack scenarios including some false positives & noise events • Attack Symptoms for each specialization (E.g., Dos, Malware , Phishing) identified • The expected results, interaction (between participants ) logs and attack conclusion arrived at by each team for each session User and Team Views User Screen Common Screen Events Suspicious Events Symptoms Submit Match Broadcast Legends Publish Functions Unknown Data CyberCog Simulator- Interaction CyberCogSimulator- Architecture Web Dos Specialist Services Microsoft POCO’s IIS Intra/Internet Malware ADO.net Specialist Controller & Model Tier Phishing View Tier Specialist Client Tier Database Conclusion • There are current gaps and limitations in Cyber Situation Awareness • Cyber situation awareness by teams involves the coordinated perception and action in the face of a change in the cyber situation • CyberCog will allow the MURI team and others to better understand team-based cyber SA and to test algorithms and tools developed for improving it Team Cognition Research Program UAS Field Data 2990 2980 Cumulative Speaking (s) Testbeds: Empirical 2970 1) UAS C2 2960 Studies in 2950 2) Navy Strategic Planning Testbed 2940 3540 3560 3580 3600 3620 3640 3660 3680 3700 Time (s) Measures ACT-R Model of Theory Development Synthetic Dynamical Systems Modeling Teammate
"IA Cooke May 2010.ppt - Wikispaces"