Methods and Metrics for Analysis of Sensemaking

Methods and Metrics for Analysis of Sensemaking Dr Karen Carr & Mr Barry McGuinness BAE SYSTEMS Advanced Technology Centre Our objectives • For this meeting: – Contribute what makes sense to us, in our given context and with our goals • In our work: – – – – Develop the ability to supply C3I „capabilities‟ (in partnership) Systems Engineering of socio-technical systems Driven by need to deliver usable and demonstrable results Science as well as engineering and domain expertise NB We want to ensure that human issues drive the developments - but we don‟t want to forget that we have to inform technology (as well as organization, process) What we mean by sensemaking Why we want to use this concept to try and answer the questions we need to answer • Our question: “How can we develop technology, design and manage systems which support/ enhance the human roles in defence operations?” – Significant human role is ability to adapt, respond to unexpected, creativity, play mind games, etc. Need to preserve & enhance that not interfere. – Support is needed for dealing with the unexpected, the unknown, as well as recognisable situations – Include broad System of Systems issues, developers, rapid change Sensemaking (& Situational Awareness) is a working concept to enable us to start manipulating, analysing, and measuring context, goals and human performance Why we want methods and metrics for studying sensemaking • Need to attribute effects - predict - in order to provide support. • Move from concepts to metrics to analysis to (testable) models. • Reduce subjective bias (influence of our own sensemaking, interpretation) • No existing clear metrics we can use - no absolutes 1. 2. 3. Understand how human performs, and what conditions facilitate „good‟ performance (what hinders) Identify the properties of organisation, process, technology, training, etc which are important for success Develop design and management methods and tools to enable implementation • NB not necessarily numbers - could be properties Range of methods • observation (non intrusive) • subjective investigation (e.g. ethnographic, knowledge elicitation) • storytelling/anecdotes (knowledge building) • metaphor (pattern matching) • scientific method (empirical hypothesis testing) • mathematical analysis (baseline) Methods and Metrics • Concepts • Metrics • Some analyses • Implications for sensemaking Concepts • Orientation – – – – – complex, uncertain situations SA determines capacity to decide and act sensemaking determines SA cognitive processes are intrinsically goal-directed people form nested hierarchy of processes & outcomes • Objectives – Understand SA and sensemaking – Feed into design & development of information systems and human systems – Applied research -- theory into practice What is a Situation? • A situation is a pattern in state space, especially one which appears to deviate from a “normal” intended or expected pattern. • Example:Aircraft fuel level aircraft fuel x time into flight Normal takeoff Unexpected rate-we have a situation! Normal cruise Time into Flight Unrecognized Patterns • An unrecognized pattern demands attention. Attention! perception comprehension ??? Perceived pattern Known patterns? Unknown pattern? Definitions Knowledge: = capacity for “action” Situational Awareness: = dynamic “situated” knowledge, i.e. capacity to act effectively here & now in a given specific situation • doing • saying • thinking Sensemaking: = process of creating effective SA in situations of uncertainty “Knowing what‟s going on so you can figure out what to do.” Situational awareness Dynamic mental representation of the current and future state of one‟s domain of action – includes awareness of • • • • • • environment entities events processes actions others‟ perceptions & intentions Through a continuous process of situation assessment – insofar as these are relevant to • performing an action, or • choosing a course of action Situational awareness SA is based on ... • prior KNOWLEDGE – SCHEMAS: generalized patterns representing typical situations – based on experience, training, culture • recent INFORMATION – direct perception of the environment – perception of instruments and displays – received communications KNOWLEDGE SA INFORMATION instruments communications PHYSICAL DOMAIN Central role of SA • SA both informs and is informed by • sense-making • decision-making Sensemaking Decisionmaking SA COGNITIVE DOMAIN Information acquisition Action performance PHYSICAL DOMAIN Inside SA: Cutting up the cake Observed Models Abstract (generalized patterns) situational schemas e.g., “Fuel leak?” “Faulty sensor?” Implied Projections mental simulations e.g., “Risk of not reaching destination” Concrete (situation-specific) Information specific propositions e.g., “rate of fuel loss is high” Intentions selected actions afforded by situation e.g., “Contact ATC and inform” Processes involved in SA PERCEPTION Acquisition of information about the given situation Diagnostic interpretation of the given situation Prognostic simulation of future situations and their possible outcomes Selection of actions to direct the given situation towards the desired outcome information COMPREHENSION models PROJECTION projections RESOLUTION intentions … All serving to support dynamically effective action Sensemaking and SA COMPREHENSION PROJECTION Models Sense making Information PERCEPTION Sensing Projections Decision making Intentions RESOLUTION Acting Sense-making: when comprehension is uncertain Decision-making: when resolution is uncertain Metacognition • Defined as: – “Thinking about thinking” or “knowledge about knowledge” – i.e. “Awareness of your own SA” • noticing uncertainties, gaps, conflicts in your mental reps • identifying information needs • employing strategies for sensemaking & decision-making ? “It’s like looking over your own shoulder.” SA Gives a subjective sense of SA SA and metacognition • Four possible states: Actual awareness: True SA False SA Confident in SA Subjective attitude Not confident in SA Appropriate Inappropriate Confidence Confidence (ideal state) (danger state) Inappropriate Appropriate Sensemaking Sensemaking Need for sensemaking Team SA and shared SA • Not the same thing • Team SA = sum of current knowledge held across a team, irrespective of who has it • Shared SA = those parts of the team SA that are common between team members Team SA Personal SA Shared SA What to share, with whom? • The nature of SA in groups is dictated by goals • Goals are hierarchic • Top-level goals are shared by all members – therefore need shared SA with respect to that objective • Lower-level goals are specific to individuals – therefore need personal SA with respect to own task • Sharing one‟s SA is necessary only to the extent that the knowledge has bearing on the goals of others Team SA and shared SA • Shared SA elements can be differentially allocated resolution comprehension perception perception perception comprehension projection projection perception resolution perception perception resolution resolution Distributed SA in the C2 HQ u Models (COMPREHENSION) Metacognition? z Projections (PROJECTION) M Information (PERCEPTION) mm m m Ops Signal Intel Intentions m m (RESOLUTION) Commander So... • Explicit sensemaking processes are needed when comprehension cannot easily occur • Sensemaking requires metacognitive awareness of own knowledge -- uncertainties, gaps • Metacognitive assessments can be wrong and lead to inappropriate subjective attitude -- and inappropriate behaviour Measuring SA • COGNITIVE approach – queries about the situation – Reveals mental reps • Multiple choice (SAGAT) • True/False (QUASA) • Sit Reps • OBJECTIVE approach – behavioural & physiological correlates – Reveals changes in metacognitive state • EEG, fMRI • Eye pointing • SUBJECTIVE approach – self-ratings of SA – Reveals metacognitive state • Unidimensional (SARS) • Multidimensional (SART) • Multidimensional and intelligible! (CARS) As a rule, take both cognitive & subjective measures together. CARS • Crew Awareness Rating Scale • a subjective tool to elicit operator‟s subjective sense of SA • multi-dimensional • generic, adaptable, easy to use Dimensions Knowledge Perception Processing Comprehension Projection Resolution Eight CARS questions knowledge Would you say you have a good sense of … 1. the most recent information 2. what is really going on here 3. what could happen 4. what actions should be taken processing Would you say it is easy for you to … 1. monitor the flow of information 2. understand the big picture 3. predict how it is likely to evolve 4. decide what actions to take Six possible responses For sure? Certain YES Definitely Definitely not Uncertain Think so Think not Don’t know Don’t need it Do I ? NO CARS results Def CONTENT Prob Prob not Def not DK NA • Perception • Comprehension • Projection • Resolution PROCESSING ||| | | |||||| ||| || ||| |||| || ||| || ||| || || | || |||| ||| | | | | | • Perception • Comprehension • Projection • Resolution || ||| |||| | || CARS results Comprehension knowledge over time 100 1. Definitely Probably Probably NOT Definitely NOT % of ratings 80 60 2. 3. 4. 40 20 0 Run 1 Run 2 Run 3 Run 4 QUASA • Quantitative Assessment of Situational Awareness • a probe tool to elicit operator‟s actual SA • mathematical : based on SDT • still under development, but promising QUASA • Signal Detection Theory Square? YES! perception discrimination • Targets vs non-targets • Hits, False Alarms, Good Misses, False Rejections • Also applies to internal (mental) representations QUASA • “Is this item true?” – Confidence in perceived truth value of items varies Number of items Max SENSITIVITY = ideal SA FALSE items Weak TRUE items Strong Confidence in truth value of items QUASA No sensitivity, poor SA Number of items Literally can‟t tell the difference between true & false items… They have similar-strength levels of confidence Weak Confidence in truth of items Strong QUASA Deception Number of items Max NEGATIVE sensitivity = worst case SA TRUE items Weak Confidence in truth value of items FALSE items Strong QUASA SA‟ Number of items Some positive sensitivity Low positive bias (acceptance threshold) Good rejections Good acceptances Bad rejections Bad acceptances Weak IB‟‟ Confidence in truth value of items Strong QUASA Example probe: “ The tanks adjacent to bridge are enemy ” Response: YES (accept as true) or NO (reject as false) Evaluate: Sensitivity (discrimination of true/false situations) = SA‟ Bias (probability of item acceptance/rejection) = IB‟‟ QUASA +100 Maximum negative sensitivity: the wrong situation! Maximum positive sensitivity: ideal SA Maximum positive bias: too rash Maximum negative bias: too cautious TYPICAL 0 -100 QUASA +100 0 Resolution: Perception: Comprehension: Projection: CoA intention model of Future information situation developments -100 Dynamic SA - D 100 80 60 40 20 0 -20 -40 -60 -80 -100 Last turn of block 4 5 6 7 8 9 10 11 12 13 14 15 16 17 SA' score Info bias (IB'') QUASA • Mathematical assessment of SA • Needs the truth! • SA, bias, components, temporal • ? Team & shared SA Behavioural correlate of SA Tracking eye-point-of-gaze (EPOG) Do EPOG patterns correlate with SA? EPOG research touch panel (systems control display) CRT CRT Nav controls SPD PR ALT ALT HDG VS CRT A/P on/off CRT Display Control Unit sidestick priority mode control panel (Flight Control Unit) PFD ND & EFIS ECAM (engines) ND & EFIS PFD Comms control unit ECAM (systems) spoilers & flaps indicators LDG GEAR controls sidestick FMS touch panel (optional for electronic charts, datalink) spoilers throttles flaps „Entropy‟ = known loss of SA COCKPIT LAYOUT forward view SA and flightdeck automation Radio “party line” Heathrow control this is Speedbird five five, descending now to flight level one four. Speedbird five five, Heathrow control, roger, descending to flight level one four. Heathrown control this is Delta four zero four, flight level two zero, request descent clearance. Delta four zero four, Heathrow control, what is your present altitude? ... Collision avoidance system +01 SA and flightdeck automation With automation 50% Reported aircraft Conventional 25% 0% Detectable aircraft Non-Detectable aircraft Traffic Situation Reporting SA and C2 digitization - ISTAR Enemy positions Own force positions BGHQ crewstation Common Operational Picture SA and C2 digitization - ISTAR Synthetic environment Battlespace digitization demonstrator SA and C2 digitization - ISTAR • 2-hr ISTAR recce operation • Performed with voice AND digital C2 systems Measures taken of mental workload & situational awareness SA and C2 digitization - ISTAR digital DEF voice PROB PROB NOT DEF NOT PERCEPTION COMPREHENSION PROJECTION RESOLUTION aspects of SA (knowledge of enemy) Some implications Both actual SA and subjective sense of SA affect decision-making & performance Technology can affect SA for better or worse Analyses with metrics provide specific insights Other work • DS1 trials • BattleLab trials • Cognitive modelling – COGNET in C2 environment – Ideal Decision Maker Building Industry-MoD partnership • ? Can be used to predict dips in SA and sensemaking needs Implications for sensemaking • Thinking about thinking • Concepts : sensemaking as processes supporting SA • Role of metacognition : group context • Metrics of SA : can be used to evaluate sensemaking solutions • Data can feed development of predictive models • Knowing what‟s going on so we can figure out what to do!