Document Sample
Chris Forsythe, Michael Bernard, Patrick Xavier, Robert Abbott, Ann Speed, & Nathan Brannon Sandia National Laboratories Albuquerque, New Mexico Research by Sandia National Laboratories (SNL) is currently being conducted that seeks to embody human-like cognitive capacities in machines by transforming the human-machine interaction so that it more closely resembles a human-to-human interaction. This document reports on the initial phase of research and development by SNL in creating a capability whereby a machine-based cognitive model provides a realtime awareness of the cognitive state of an operator. In the capability referred to as “Discrepancy Detection,” the machine uses an operator’s cognitive model to monitor its own state and when there is evidence of a discrepancy between the actual state of the machine and the operator’s perceptions concerning the state of the machine, a discrepancy may be signaled. The current project offers successful evidence that a machine may accurately infer an operator’s interpretation of situations based on an individualized cognitive model of the operator.

SCOPE OF RESARCH Our research is based on an individualized cognitive model of an operator’s knowledge for a specific task domain (Forsythe & Xavier, 2002). We utilized the cognitive framework previously developed by SNL to computationally model an operator’s cognitive processes. This framework, shown in Figure 1, was partially inspired by naturalistic decision making (Klein, 1997; Klein, Calderwood, & MacGregor, 1989) and oscillating systems theory (Klimesch, 1996). The model emphasizes the processes whereby an individual recognizes situations on the basis of patterns of cues within the environment. In its simplest form, the model consists of (1) a collection of situations, (2) cues associated with those situations, and (3) mathematical expressions for the recognition of situations based on patterns of cues. In a functional cognitive model, streaming input from the task environment feeds perceptual algorithms. These algorithms operate in parallel and, depending on the corresponding cue in the associative network, provide either a continuous or discrete level of activation. For a given perceptual algorithm, if the level of activation is sufficient, a response will be elicited from the corresponding node in an associative network. The associative network is modeled as an oscillating system wherein each node is a separate oscillator (cf. Klimesch, 1996). Perceptual activation causes corresponding nodes to begin oscillating with activation spreading to other nodes for which there exists an associative relationship. Spreading activation between nodes in the associative network may be sufficient to cause a node, receiving

activation, to begin oscillating or may prime the node such that the node exhibits a lowered threshold for activation by perceptual processes. Situation recognition occurs when there is activation of a pattern of nodes in the associative network that corresponds to a given situation. Computationally, recognition is based on an evidence accumulation1 process in which cues differentially contribute to or subtract from the evidence for different situations, and the evidence for each situation is periodically (i.e., every 250 msec) updated providing an ongoing level of evidence for each situation. At any given point in time, multiple situations may have sufficient evidence for their recognition (i.e., multi-tasking). Implemented in systems for augmented cognition, a customized model is developed to reflect the knowledge of a specific operator as the person interacts with that system. As shown in Figure 2, operating in real-time, this model is used to interpret the ongoing state of the system. To the extent that the system has accurate and timely data concerning its own state, and the operator on whom the model was based has a fundamental understanding of the system and associated tasks, this serves as an idealized cognitive processor in that it has an essentially unlimited capacity to process multiple cues in parallel.


The evidence accumulation approach to situation recognition is based on a synthesis of findings from the EEG literature that included Coles, (1995); Kok (1990); Kounios, (1994); & Wilding (2000).

Discrepancy detection is accomplished using real-time data from an operator. The system monitors the operator’s actions, or lack of action. For each situation, typical actions have been identified. Discrepancy detection is triggered when an operator either commits an action that is inappropriate given relevant situations or an excessive period of time transpires in which an operator does not execute an action that is expected given relevant situations. Further development of the capability for discrepancy detection will utilize additional sources of input such as eyetracking and physiological indicators associated with situation recognition processes. METHOD Task Materials. The Distributed Dynamic Decisionmaking (DDD) Airborne Warning and Control System (AWACS) simulation trainer was chosen as the task environment for prototype development. This environment offered a reasonably high-level of complexity in which it was necessary for operators to multi-task as they simultaneously controlled multiple assets and responded to multiple threats. Participants practiced until they achieved a high level of expertise in playing the game. The criterion for expert performance consisted of having the knowledge and skills to consistently play the game without the loss of friendly aircraft due to lack of fuel or enemy missiles, without enemy incursion within the friendly zone, as well as the destruction of all enemy jets (see Figure 3). The Cognitive Model. Research has shown that with dynamic environments such as the DDD-AWACS task, individuals tend to have distinct cognitive models that are tied to their skill level, as well as their past experiences with the task environment (e.g., Goodrich & Boer, 1999). Therefore, a key question concerned whether the differences between individual cognitive models are sufficient to necessitate development of individualized models. A comparison of cognitive models of two operators with equivalent expertise demonstrated that individualized cognitive models are required for systems to perform reliably in that only 26% of the total cues and situations associated with the cues identified by the two participants were used by both participants. Consequently, individualized cognitive models were used in our design. In order to populate each cognitive model with expert knowledge, we utilized an individualized knowledge elicitation method developed at SNL that was designed specifically for this cognitive framework. This method was explicitly designed to acquire both explicit and implicit knowledge. Discrepancy Detection. A method of Discrepancy Detection that is based on the actions expected under various conditions (i.e., situations) was implemented. The general idea is that in the context of a particular situation instance, it is expected that an operator will perform certain actions— “what needs to be done”—with respect to the entities in the situation instance. The pattern of action expectations is derived from the knowledge elicitation process. The

Discrepancy Detector module takes as real-time input, the current actions and situation instances inferred by the cognitive model. RESULTS Assessing Expert Performance Accuracy. Formal testing evaluated the accuracy of the cognitive model for interpreting situations within the task environment, relative to the interpretations of the actual participant on whom the cognitive model was based. For the evaluation, a session was used in which one of the participants obtained a perfect performance score. The session was recorded so it could be played back readily. To create a reference against which the situation interpretation of the model could be assessed, the participant repeatedly reviewed the recording indicating points in time when situations became active and subsequently, when situations were completed. The results revealed a system capability to infer an expert operator’s ongoing cognitive interpretation of the state of the system with an accuracy of 87%. Moreover, only considering the occurrence of situations, as opposed to recognizing when situations were not preset, the cognitive model was 91% accurate. There was a 10% incidence of false positive and 3% incidence of false negatives. Assessing Novice Performance Accuracy. As a further evaluation, a second reference was created in which the participant deliberately performed poorly. Enemy jets were allowed to destroy two friendly jets, and two jets were purposely aloud to run out of fuel, and thus were destroyed. In addition, one enemy jet was allowed to pass through the friendly zone, thereby lowering the defense score. The cognitive model performed equally well in this condition, suggesting that the prototype system is relatively robust. DISCUSSION This project demonstrated that a machine can accurately infer in real-time the cognitive processes of an operator using an individualized cognitive model of the operator. This provides the foundation for a variety of concepts for augmentation that rely upon a capability for a machine to accurately model the cognitive processes of an operator. The current project illustrates one manifestation of this capability. Through discrepancy detection, the machine not only detects that the operator has committed an error, but the machine recognizes the nature of the error within the context of the operator’s overall knowledge and understanding of the system and associated tasks. Consequently, the machine may interact with the operator in a manner that is substantially more meaningful than the familiar error messages presented by current software applications and operating systems. Furthermore, to the extent that there exists deficiencies in the operator’s knowledge of the system and associated tasks, there is a

meaningful basis for training intervention to correct deficiencies in the operator’s knowledge. The results obtained through the current project have broad application. An immediate next step concerns the integration of the prototype demonstrated here with other sources of input to provide the machine with a more comprehensive ongoing representation of an operator. Initially, plans are to explore eye-tracking in addition to operator actions as one source of input. Whereas actions provide explicit data concerning an operator’s interpretation of situations, eye-tracking offers a more subtle indicator. To the extent that recognition of a situation requires that an operator process visual cues within the environment that are associated with the situation, data indicating visual cues that most likely have or have not been processed should enable somewhat better predictions. Another comparable approach to improving machine awareness of an operator is to provide input from electrophysiological and brain imaging sources. Efforts have begun to identify measurable physiological correlates to the cognitive processes modeled in the Sandia cognitive framework. Again, these data offer potential for systems to more accurately model the ongoing cognitive processes of an operator. However, the benefit should go two ways. Where physiological measures are employed to derive a real-time indication of workload, these indications may be enhanced by input supplied by a cognitive model. For example, the cognitive model may indicate that there are six different situations that characterize events at a given point in time. Such data could then be used as a modifier for physiological measures (i.e., if the cognitive model indicates that workload should be high, this serves to confirm physiological measures suggesting that workload is high). ACKNOWLEDGEMENTS This work was performed at Sandia National Laboratories. Sandia is a multi-program laboratory operated by Sandia Corporation, a Lockheed-Martin Company, for the United States Department of Energy under Contract DEAC04-94AL85000.

REFERENCES Coles, M. G. H., Snid, H. G. O. M, Scheffers, M. K. & Otten, L. J. (1995). Mental chronometry and the study of human information processing. Electrophysiology of mind: Event-related brain potentials and cognition, Oxford: Oxford Science Publications, 86-131. Goodrich, M. A., & Boer, E. R. (1999). Multiple mental models, automation strategies, and intelligent vehicle systems. International Conference on Intelligent Transportation Systems Cat. No. 99th 8383, 859-864. Forsythe, C., & Xavier, P. G. (2002). Human emulation: progress towards realistic synthetic human agents. Proceedings of the 11th Conference on Computer Generated Forces and Behavioral Representation, Orlando, Florida. Klein, G. (1997). An overview of naturalistic decision making applications. In C. E. Zsambok & G. Klein Naturalistic Decision Making, Mowah, NJ: Lawrence Earlbaum, 49-59. Klein, G., Calderwood, R., & MacGregor, D. (1989). Critical decision method for eliciting knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19, 462-472. Klimesch, W. (1996). Memory processes, brain oscillations and EEG synchronization. International Journal of Psychophysiology, 24, 61-100. Kok, A. (1990). Internal and external control: A twofactor model of amplitude change of event-related potentials. Acta Psychologica, 74, 203-236. Kounios, J., Montgomery, E. C., & Smith R.W. (1994). Semantic memory and the granularity of semantic relations: Evidence from speed-accuracy decomposition. Memory and Cognition, 22(6), 729741. Wilding, E. L. (2000). In what way does the parietal ERP old/new effect index recollection. International Journal of Psychophysiology, 35, 81-87.

Knowledge Engineering
Cues and Knowledge of Ongoing Events Associative Knowledge
Pattern Recognition

Situational Knowledge
- Situation A - Situation B - Situation C -

Cues in the environment activate concepts in associative net Patterns of activation are recognized that correspond to known situations

Interpretation of Situation

Situations and corresponding knowledge

Knowledge of appropriate actions (e.g., scripts) are implicit to situation recognition

Figure 1. Framework for Computationally Modeling Individualized Knowledge and the Cognitive Processes Enabling Situation Recognition

Mouse/Joystick, Visual Fixation, Communication, Physiological Signals

Environment & Machine State

Discrepancy Detection


Inferred Operator Situation Model

Figure 2. Example of a System a Using Machine-Based Cognitive Model of an Operator for Augmented Cognition. In this Case, the Objective is for the System to Detect Discrepancies between the Actual State of the System and the Operator’s Perceived State of the System (i.e., Discrepancy Detection)

Friendly zone

Enemy jet Friendly bases Indicates time to complete the action in progress

Friendly jet

Status reading subject

Report Window: Summary of all incoming messages.

Confirmation Window: Event log listing actions taking place in the region of interest (and by whom)

Figure 3. The Screen Layout of the DDD-AWACS Task Environment.

Shared By: