Virtual Human IQ: Near Term Support for a Human Participant’s Perception of
Increased Virtual Human Intelligence
Randy Boys
Raytheon, Network Centric Systems
rboys@raytheon.com
ABSTRACT Second Life™, using only a keyboard or digitized speech
as input.
This paper proposes a number of short term ‘remedies’
for increasing the robustness and interactive intelligence In avatar interactions with VH, the limited „intelligence‟
of a Virtual Human populated Virtual world, particularly of computer-generated entities rapidly compromises the
for team-based human immersion in military training responsiveness of the VH and the overall quality of the
environments, specifically Infantry small unit „interpersonal‟ engagement. Although this is less
engagements across a wide range of operations, including noticeable in the dominantly kinetic (i.e., ID-and-shoot)
the solicitation of leader/Soldier decision-making in interactions typical of the popular „first person shooter‟
complex non-kinetic engagements. gaming environment, the limited VH capability rapidly
compromises the ability to use VH in more complex
KEYWORDS: Virtual Humans, Simulation Immersion, scenarios. Just as the holodeck is the grail of immersion
Infantry Training, Small Unit Training into a Virtual world, the Turing Test [2] would be the
Holy Grail of human VH interaction. The Turing
Test is a proposed setting in which the human participant,
1. INTRODUCTION screened from direct observation of the „tested‟ entities,
cannot readily ascertain whether the participant in their
Technology that introduces individual humans vs. human- conversation is another human, or an „intelligent‟
operated platforms into Virtual space is increasingly computer.
capable and commonplace. Applications of this
technology range from the creation of an animation- There are many technical pursuits being applied (by a
supported transaction using mobile devices, to role-player growing commercial industry) to increasing VH
representatives in massively multiplayer gaming intelligence. These include speech recognition, natural
environments, to team-based, full-body motion capture in language processing, self-adaptive (i.e., „learning‟
real time. Although this latter capability has not yet systems, generally applicable only where a sufficient
attained the Holy Grail as set forth in the holodeck of Star training base exists), goal-directed behaviors and other
Trek™ fame [1], it is clear that humans will increasingly reasoning mechanisms, terrain/threat trafficability and
„meet‟ in Virtual space, and that the meetings will be with elements of „mission planning‟, group and team-based
both avatars and with Virtual Humans (VH, also as „AI‟ behaviors, etc. In the human-immersive Virtual space,
for Artificial Intelligences). almost all approaches that demonstrate VH „IQ‟ boil
down to a matter of VH responsiveness to the actions of
In these applications, human-to-human (i.e., avatar-to- the human participant(s), and the inferred intelligence
avatar) interaction is limited by the instrumentation of the demonstrated by that responsiveness.
avatar-creating human(s) … capture of speech/text,
capture of gestures, and a priori capture of the human‟s There are means other than programmed VH intelligence
appearance … but is also limited by the depiction of those by which the perception of intelligence can be influenced.
avatars in the Virtual space. Even the crudest of these This paper proposes a number of short term „remedies‟
avatars can, however, readily exhibit the „intelligence‟ of for increasing the robustness of a VH-populated Virtual
their controlling human … such as during the exchange world, particularly for team-based human immersion in
between two avatars at an information kiosk supported by military training environments, specifically Infantry small
unit engagements across a wide range of operations,
including the solicitation of leader/Soldier decision- computing, particularly widespread personal computing
making in complex non-kinetic engagements. (most recently including mobile devices and gaming
consoles) and (2) whole body immersive individual/team
2. VIRTUAL HUMANS DEFINED simulators, „immersion‟ referring to the capture of body
dynamics to control the Virtual representation of the
In the conventions of the military simulation community participant.
(training, wargaming, ops analysis, and materiel
development), VH is a Constructive simulation. VH Although both domains create an avatar of the participant,
actions are computer-generated. An avatar ... an entity the avatar in the first case may be as little as a state
representing a human controller …is a Virtual simulation. representation to indicate
That the VH exists in Live or Virtual space has caused „speaking/typing/touch[screen]ing‟ vs. „waiting‟, imposed
them to be termed Virtual Humans. [3] And, although the on an animated picture of the participant or the selection
label AI is common for VH in the gaming/entertainment of other graphical representations from an available
industry, the term AI in gaming also applies to platforms library of characters. In gaming systems, simple
or other lifeforms, not just to humans (i.e., „VH‟ and this movement and object control states are also added to this
definition of the „human‟ AI are equivalent). Similarly, animation, these being controlled by keyboard, mouse,
Artificial Intelligence, the sub-discipline of computer speech, touchcreen/active digitizer, or specialized devices
science (interdisciplinary with regard to at least like gaming props and Kinect™-like video recognition.
psychology and mathematics), is commonly referred to as In the second type of avatar generation, whole body
„AI‟ but is not the same as „an‟ artificial intelligence (AI) immersion, control of the avatar in immersive simulation
as the VH or platform entity. is by more natural (and robust) means, and has attempted
to meet the military adage of „Train as you Fight‟. The
The term VH is therefore used here to describe all non- end state of this latter development would be similar to
avatar human entities presented in Virtual space, despite the capability set forth as the holodeck of Star Trek fame,
the possible confusion resulting from the Constructive noting that there was no avatar creation on the holodeck
nature of VH generation. (Live participants entered the Virtual space). In this end
state, the trainee(s) would enter the Virtual training space
Also note that both military and academic simulation with their objective tactical gear and „just do it‟.
disciplines have coined other conventions for topics
related to VH, and that the boundaries between Live, A similar non-avatar type of immersive Virtual training
Virtual, and Constructive are challenged when the does exist in our non-Star Trek world, this being the case
simulated (or enhanced) entity is a human. All instances where Live trainees interact with Virtual entities. The
of „AI‟ in the gaming definition are equivalent to VH technologies with regard to creating the perception of
Computer Generated Forces (CGF) as defined in the „intelligence‟ are identical to those in the Virtual world.
context of military simulation, all being Constructive It could also be held that this Live environment is
simulations. The term Semi-Autonomous Forces (SAF) arguably the proper end state of this type of immersive
refers to a subset of CGF that accepts run-time input from training, although it will be constrained by the inability to
a human controller, but that input is not conducted by replicate necessary exercise-specific influences in that
assuming first person control of the CGF (e.g., detailed Live world (at least, that is, until the matter-bending
motion, sensor/weapon control, etc.). In the gaming holodeck technologies exist!). These hybrid and
industry, and increasingly other practical applications augmented realities are not described here.
(such as training), the disciplines involved in creating
„Artificial Life‟ are numerous, and expanding rapidly at The first domain, physically non-immersive simulation,
both academic and application levels. benefits from the rapidly growing commercial
entertainment industry, although it is increasingly
buttressed by related telepresence industries (e.g., virtual
3. IMMERSIVE TRAINING FOR meetings, training (including VH trainers), telerobotics,
INFANTRY (AND RELATED) SMALL telepresent command/control). Furthermore, the motion
UNITS capture and speech recognition capabilities of gaming
consoles, along with the addition of props for interaction
Only recently has the training of individuals in Virtual with the Virtual world (generally as weapons) have
space become practical, „individuals‟ being walking, significantly increased the immersive nature of these
talking humans, as opposed to representations of a man- systems.
machine platform. This has been brought about by
advances in two domains: (1) no-motion or limited The latter domain, team-based trainee immersion in
motion immersion, enabled by affordable/capable Virtual space via specialized instrumentation, continues to
exist as a high end military or specialized industrial include cost avoidance (e.g., real role players take time
application. Features of these systems include full 3D and money to train, coordinate, and utilize; using Soldiers
motion capture, including gross motion and fine gesture. in this role is also suboptimal), session
Although the experience shares many technical attributes standardization/replicability, and the ability to present
with the increasingly interactive gaming applications, the kinetic or socio-cultural settings that are otherwise
intended training capabilities continue to differentiate difficult, dangerous, or expensive to replicate. In a
them. [4] training exercise emphasizing leader development, the
ability to use VH as the subordinates would also be
In the intended military team training application, various beneficial.
immersive elements are critical to the training experience,
not all of which is oriented to the transfer of procedural Unfortunately, the state of the art of Virtual roleplayers
knowledge. The immersion attempts to be as realistic as limits their applicability in immersive simulation.
possible in emotional, perceptual, kinetic, and cognitive Development to make them compelling is expensive and
dimensions. This is not to say that increased simulation scenario-specific, and even then is presented in a medium
fidelity is equivalent to increased training capability, but that is only nominally interactive, let alone immersive.
there are elements of the simulation‟s replication of the They are also only in their infancy with regard to being
range of military operations that must be attained if leader able to react to any session-to-session variations of the
development and team coordination are to be supported. trainees or the session events/geometries. The application
Note that these skills are not all physical or cognitive in of „reasoning‟ VH in narrow contexts has been achieved,
nature (e.g., leadership development, situational but not with regard to the general application within the
awareness). Similarly, stress inoculation … ensuring that now-available small unit immersive simulation. That is,
a Soldier‟s psycho physiological readiness for their first the avatars have entered the space, but the space remains
firefight is supported by (and no worse than) their last unpopulated. Similarly, the vector in gaming applications
simulation … is a goal of such „high end‟‟ immersive has emphasized the kinetic interplay of combatants (and
experiences. [5] These degrees of warfighter immersion their appearance), a dimension along which avatars and
have been largely attained in platform simulator training, VH do not vary greatly.
but are only recently available (and still emerging) for the
individual Soldier, small unit, and small unit leadership. Ultimately, each element of human socio-cultural
Unlike the platform simulations, the capability of the VH behavior that is not supported by the trainees‟ interactions
that populate this type of immersive space is critical to the with the Virtual roleplayers in the immersive Virtual
experience. space (to include the physical presentation medium) can
contribute to lowering the trainee participants‟ overall
For the purposes of this paper, the term „immersive interaction acceptance threshold (i.e., their „suspension of
simulation‟ refers to the extent to which contemporary disbelief‟), against which skills development may (or may
Virtual simulation can replicate this desired Infantry small not) occur.
unit „walk in‟ end state, noting, however, that immersion
can occur in any type and fidelity of simulation, and that Our recent experience with full-body, full team, real-time
the gaming industry would welcome the widespread immersion in Virtual space [7] indicates that the actions
availability of a gaming console variant of the holodeck! (and reactions) of virtual roleplayers are more important
to the trainees‟ degree of immersion than is the
4. WHY VIRTUAL HUMANS? appearance of these roleplayers, or the overall functional
bandwidth („intelligence‟) of those roleplayers. For
In the immersive space, the training goal is to both allow example, a Virtual character with precise facial
the team to jointly hone their individual and team skills expressions or other high fidelity rendering, or one that
but, just as importantly, to introduce thinking, reactive recognizes a large vocabulary, is … if a trade must be
adversaries and other human elements to the otherwise made … less important to the training experience than are
dominantly maneuver and weapons-based Live exercises. roleplayers that the trainees perceives as reacting to them
The changing nature of small unit operations … in an immersive, real-time, fashion.
peacekeeping, humanitarian, disaster response, embedded
insurgents, coalition forces, etc. … requires this wide It is as easy to list the characteristics of compelling
variety of human-based interactions, and even roleplayers human-to-human interaction as it is to note the limitations
in Live training have extreme difficulty approximating of VH. Indeed, the lists are the same:
these conditions, and at high cost. [6] Verbal communication (understanding, not
„speech recognition‟)
The goals behind using Virtual roleplayers in small unit Verbal communication in lieu of shared language
(i.e., Fire Team and Squad) Live training exercises
Gesture recognition (overt) and „body language‟ VHs can include the introduction of a human controller of
(gross, as in posture or use of space, or fine, as in the VH.
hand motion or facial expression)
Perceived intelligence (a „thinking‟ The use of humans to control simulation-generated
friend/foe/neutral, exhibited independently, or in entities in simulation training is not new. In the
reaction to other VH state, or in reaction to Constructive arena, these roleplayers are often called
trainee state) „pucksters‟. The change needed for VH control, however,
These features of „intelligence‟ are perhaps obvious, but is related to the types of cueing the controller requires,
are often neglected as attempts are undertaken to add often calling for near-Virtual perception while immersed
„higher level‟ skills (e.g., „reasoning‟, display of affect) to in the environment, and also the nature of the VH
the VHs. Indicators of „purposeful‟ behavior (i.e., behavior, often non-kinetic, that is desired to be
reactive and seemingly intelligent) include, for example: introduced. Current individual player/roleplayer tools
such as VBS2 allow a user to generate SAF, or to control
I walk in to a room and you look at me (or
an entity as a user-controlled avatar. Entertainment
another appropriate cultural response); I point at
gaming systems do the same, including rudimentary
someone, you look at them; When we talk, you
may move towards, or away from, me control by body position, gesture, and speech. Recent
studies, such as those at the USMC Warfighting Lab at
In group-on-group settings, the dominant actors Quantico, have assessed the ability of operators to control
in each group position uniquely (to each other multiple computer-generated entities, and have achieved
and to their subordinates), and their actions acceptable performance in ratios of 1:5-1:10 „controlled‟
overtly and covertly control the actions of other virtual humans [8], with the operator assuming avatar
members of their group
control of „needed‟ assets when deviation from scripted or
I point my weapon at you, you take cover or otherwise limited VH behavior is required. However,
react otherwise … immediately most of these successes are in the population of traditional
I talk or call to you and you orient to me, or react kinetic engagements typical of the documented battle
otherwise drills, as opposed to excursions into the human dimension
The use of space is deliberate: in kinetic of increasingly diverse and autonomous small team
situations, geometries are optimized and risk is operations.
avoided; in non-kinetic situations, evidence of
purpose may be as simple as how and when I In this reminder of the desired immersive Virtual training
cross a clearing or move from behind a table, time, the paradigm for building SAF platforms and
when I put down a piece of paper, etc. kinetic-capable gaming adversaries/teammates needs to
change [in the author‟s opinion] in emphasis from adding
If I am busy with a task, you (usually) would not
prompt me to begin another task tactical functionality or visual fidelity to, rather, the
addition of a number of available „human CGF‟ specific
Behavior between party members is coordinated, attributes … of which the subset of Virtual training that is
and in manners the origin of which is not defined by small unit team training in a Full Spectrum
obvious to me; this infers that there is a shared Operations context is almost 100%, given the current state
perception or understanding between these other of VH intelligence/ responsiveness in these contexts.
intelligences
Only in shoot/no-shoot situations do most VH play their
role well, at least in free play small unit team training. Of
5. THE HARD PROBLEMS, AND HOW TO course, as VH „AI‟ capability or visual fidelity increases,
AVOID THEM it would be added, too.
First, the bounding box. The „quick fix‟ Many of these „quick fix‟ solutions are just „checks‟ to
recommendations that follow are not addressing those make sure that a VH‟s dumb behavior is not made
contexts in which deeper VH intelligence is currently both obvious. These include:
applicable and capable (e.g., specialist procedural training, Responsiveness
potentially including perceivable evidence of VH o to me
reasoning such as natural language processing, highly o to others
detailed display of affective reactions, the ability to guide o to events
trainee actions, etc.). Immediacy of response
o At least as critical as the accuracy of
In the more dynamic, less bounded, and team-oriented response
Infantry full spectrum operations immersive training
space, a stop-gap for the limited utility of contemporary
Unpredictability from remote centers and by operators that are both SMEs
o Scripts must ensure variability in the training and in the role-playing. This would be,
Coordination however, both complex to execute and of limited cost
savings.
o A subtype of responsiveness and
unpredictability that can be Tethering VH to avatars in entertainment systems is
cognitively/computationally „deep‟, but
common practice, but then suffers from most of the
there are simple approaches/checks that
difficulties that make the VH so easy to „pick out of the
allow multiple VH to demonstrate
coordinated perception or purpose crowd‟. The more complex the situation (i.e., in other
than kinetic force-on-force engagements), the more
Motion is natural (and not obviously driven by a limited is the ability of the avatar to impart meaningful
library of oft-repeated scripts … no response is
better than the same one) independent behavior to the tethered VH.
Motion is purposeful with regard to terrain and If, however, the degree of immersion of the VH-
objects
controlling avatar in the Virtual world is high, the VH in
o This does not have to equate to on-the-fly the environment can be cued by that avatar‟s human
mission planning/replanning. perceptual abilities and reasoning (responsive actions).
This and other types of goal-directed This can be done without the VH being „tethered‟ to the
behavior are the subject of „big AI‟, avatar in the conventional Computer Generated Forces
the computational field (e.g., NP- (CGF) sense. By „degree of immersion‟, the reference is
complete plan optimization, goal- to the immediacy of the VH „controlling‟ avatar‟s (or
directed production systems, pattern avatars‟) ability to perceive the actions of the trainee
recognition, etc.), but are not the only avatars and to respond to those perceptions. Ideally, the
approaches to executing purposeful immersion would be full body, free motion, with binaural
and useful behavior based on a sound and excellent visual immersion (e.g., HMD or 360
momentary set of variables degree projection). If the controller is a typical
Preprocessing of the environment for workstation-based avatar controller, the cueing provided
purposeful consumption by the VH is by the „watch me‟ VH-controller would be more limited.
readily available, and includes not just
simple trafficability analysis, but can Control over the VH by Virtual roleplayers, who can see
include preprocessing of spatial and understand the trainees, is distributed to by both the
overlays for available avatar‟s movement and commands. Although notionally
cover/concealment, kill zones, only a single avatar is required to add the VH „perception‟
ingress/egress routes (under varying and control, there is no limit on the number of controller
constraints/objectives), and dynamic avatars, subject to the aforementioned concern regarding
modification of these preprocessed cost. The avatars can be portrayed as needed by the
zones/paths/objects given the VH and exercise (uniform/dress, race, gender, etc, even voice
avatar position/situation modulation).
So, without adding „real‟ AI to the VH so as to increase Under the assumption that the VH controller is immersed
the human participant‟s perception of that VH intelligence, in a manner similar to those now available devices that
nor adding the hard-to-do „real‟ facial, gesture, and place trainees „in‟ (not in front of) the Virtual
emotive expressiveness … all of which have a long way environment … such as the VIRTSIM, CombatRedi,
to go before they pass the Turing Test … how can we ExpeditionDI, etc. systems [9], examples of the cues that
maintain the degree of immersion that is otherwise only the controller can provide to the VH include (as
possible with interaction between only human-controlled examples):
avatars? Responsiveness, and immediacy of response:
Add cued motion or orientation (e.g., Trainee to
The simplest solution is to have human controllers of the whom my attention is focused, places that I am
VH. If, however, this is on a 1-to-1 basis, they should just looking, direction in which any required group
have been introduced as avatars in the first place, as no motion should be oriented, and even overt
cost savings would otherwise have been attained. The gestures to specific VH for actions to be taken.
possibility exists that a „service center‟ can apply a Unpredictability: The ability to task, from within
limited number of human controllers to the somewhat the simulation, specific VH promotes
predictable number of needed roleplayer avatar instances unpredictability. Similarly, the queuing of VH
in a scenario, or set of concurrent scenarios, including behavior by the controlling avatar lends to
inherent human unpredictability, as well as the interviews where timing is less critical, the
opportunity to overtly inject different VH introduction of a delay … such as those
responses. associated with the processing of affective
Coordination: Tethering to the immersed presentation or natural language processing …
controller generates one level of coordinated can still allow training to occur, but these delays
behavior, and the specific tasking of individual are probably not appropriate for infantry small
VH yet another. To the extent that it is possible team immersive training.
to „hide‟ from the trainee avatars the nature of Behaviors should not be repeated. Triggered
who is the VH-controlling avatar and who the scripts, including position/posture changes, must
VH are (including the use of subtle commands be spaced appropriately. This requires that the
the trainees may not notice), the more dynamic script be triggered with some complexity (at
the interchange with the total „group‟ becomes. least with an awareness of the prior incidence of
Purposeful motion: Without the aforementioned the script) and even possibly recursive.
preprocessing of space with regard to As mentioned above, purposeful use of terrain
alternatively weighted movement „profiles‟, it is and objects implies „intelligence‟. Much of this
likely that VH movement, even if cued or can be preprocessed and applied at runtime. The
commanded by an immersed controlling avatar, nuances of character positioning and other
will continue to „give them away‟. This can be dynamic events in those „response spaces‟
avoided by careful scripting of the scenario and (notionally as precomputed weighted space
by the controlling avatar‟s familiarity with the grids/voxels) also allows for variability in
scenario. otherwise scripted behavior.
VH should respond to each other. Much of the
Given a high degree of cued VH performance, and when cueing mentioned above for the „VH-controlling
the roleplaying immersed avatar understands the scenario avatar‟ is also available for VH-to-VH
and the VH capabilities, it is possible to conceal the coordination. If one entity stops what they‟re
avatar vs. VH nature of the entities in the environment. doing to orient to an event, a second VH that
That is, each of the non-trainee players in the scenario may not have been triggered to that event can
could be a human, or a VH. The benefit is that the trainee still be triggered by the VH response. Of course,
avatars must attend fully … at least initially (until the the nature of shared responses can vary from the
limitations of a VH are „uncovered‟) … to all entities, not simple to the complex, such as that addressed in
just the one(s) that may be thinking, reasoning humans. the literature on agent-based behavior
(cooperative and self-based) and computer-
supported cooperative work. The argument for a
In the absence of an immersed human controller (or
human-like perceptual and spatial „front end‟ for
controllers) of the VH, there are maxims to observe in initiating intelligent VH response is addressed as
populating the Virtual space if it is to become an cooperative perceptual models. [7] The
immersive dynamic decision space, demanding (and argument is made that endowing VH with a
allowing) meaningful interaction with the VH population. reasonable understanding of space, perceptual
acuity, attention, and „internal‟ filtering (object
Move when someone talks. Although mediated knowledge and context-driven attenuation) is
by cultural norms, this typically includes looking useful in producing human-like VH reactions to
at the speaker. stimuli.
Nobody stands still if a weapon is pointed at
them. The response is immediate. Hands will 6. CONCLUSION
go up, talking will start, movement to cover will
be initiated, something. Anything. Based in part on our experience interacting with other
Not limited to the cases above, VH must humans in the immersive Virtual space, which was
immediately respond to avatars. It is possibly surprisingly different than interacting with other humans
more important that the response be immediate in gaming styles of Virtual space, recommendations for
than it is that it is appropriate. And, note that a the nature of VH in this more first person and complex
late response that is reasonable, but late, is not environment emerged.
appropriate if the intent is to continue the users‟
immersion. The response does not have to be Not all behavior to impart perceived „human-ness‟ to VH
„intelligent‟ to at least demonstrate that real-time is related to intelligence. This includes not only motion
reasoning has been triggered. That is, knowing
and demonstrations of affect, but also unintelligent
that it is appropriate to do something is half the
behavior. We are, admittedly, human! It is not only
game. That said, in situations such as structured
acceptable, but possibly preferred, that VH perform … at unit, and leader performance (and training progression) in
times … in apparently incorrect manners. (This, by the the types of simulation-based training that has been
way, is one of the cited faults of the Turing Test, in that it discussed here, across both the gaming and highly
is a measure of perceived intelligence, not humanness.) immersive variants. Providing (and managing) this
training architecture is the intent behind much of the
The intent is not to argue whether a VH can imitate a recent issue of an Army pamphlet on training change
human vs. cause an observer to believe it was human, nor vectors over the next 5 years. [11]
to measure VH IQ, but, rather, to point out that the
dynamics of a VH are complicated in a different, „human‟ REFERENCES
dimension, when compared to the Virtual and
Constructive modeling of warfighting platforms with [1] The holodeck was introduced in the pilot episode of Star
which we have grown comfortable in our development of Trek: The Next Generation, although it has been
CGF, and even with regard to less immersive styles of inferred in an episode of the animated version of Star
team Virtual exercises. Trek that followed the original television series.
Similar technical applications had been envisioned in
Similarly, in an attempt to „replicate reality‟, there has earlier scientific, science fiction, and other popular
been little research on those factors of the immersive media, but none as pervasive nor as technically
Virtual space (as defined here) that are most salient to the complete as the holodeck. The following reference
provides not only the history of the holodeck, but
training intended to be performed in this space. This is in describes many aspects of immersion in Virtual space
part due to the relatively new ability to immerse teams of complementary to the discussion of VH addressed in
participants into this type of Virtual space, but also the body of this paper.
because of the many successes achieved in *other* types
of immersive environments and with other types of VH, http://en.wikipedia.org/wiki/Holodeck , Holodeck, 13
which may or may not apply, or which may need to be May 2011.
applied in a tailored manner. Similarly, the nature of
interpersonal interactions in is likely to place [2] Turing, Alan (October 1950), "Computing Machinery and
developmental demands on distributed simulation Intelligence", Mind, LIX (236): 433–460.
protocols such as LVC-IA, DIS, HLA, TENA, and CTIA. Note that the fact that a teletype was proposed as the
interface mechanism by Turing can readily be
The observations and recommendations presented here extended to the presentation of both the avatar and the
have assumed that all players are in the Virtual space. VH into Virtual space, in that the ability to present
The same mechanisms for increased avatar/VH both can be made equivalent. The degree of modeled
interaction can be introduced to Live training vs. transduced motion is otherwise the only difference,
environments in which a Virtual player/players are and can be made equivalent for the purpose of such a
introduced, whether those players are avatars or VH. In „test‟.
the case where the immersed VH-controlling avatar is in
[3] Balcisoy, S. Kallmann, M. Torre, R. Fua, P. Thalmann, D.,
both their Virtual world (beside and behind them) and in Interaction Techniques with Virtual Humans in Mixed
the Live world (in front of them, via a video feed, raw or Environments. In International Symposium on Mixed
augmented), there is very little difference from the all- Reality, Tokyo, Japan, 2001.
Virtual example, other than the types of influence that can
„permeate‟ the Live/Virtual boundary. Even that [4] Although there are many bodies of research regarding the
boundary is quite permeable, with the proper use of Virtual simulation for training, and particularly
instrumentation of the presentation area. dismounted infantry training, a particularly
comprehensive bibliography was compiled by the
A final observation: the freeplay nature of immersive Army Research Institute as their contribution to the
NATO Research and Technology Organisation
simulations must be tempered by the need to provide compendium final report regarding “Human Factors
standardized training. The flexibility of Virtual Issues in the Use of Virtual and Augmented reality for
simulation can allow for a learner-centric pull of training, Military Purposes,” as:
in which exercises may be repeated, modified, interrupted,
and replayed as needed, and with automated feedback. Goldberg, Steve, Knerr, Bruce, et al, RTO-TR-HFM-
However, the scenarios (including backstory, VH 121-Part-I (pp 7-19/28), Dec 2005 (also as ISBN 92-
behavior, etc.) must be a part of a skills and knowledges 837-1136-X)
progression, much of which occurs not only in other
simulations (Live, Virtual, and Constructive) but in [5] The intent to use immersive training for individual and small
unit training has appeared increasingly in the writings
classrooms and computer-based individual training. A of senior Army and Marine Corps leadership. Rather
significant challenge lies in the assessment of Soldier, than cite one or more of these calls for action, the
following reference provides what may initially appear Corps Warfighting Laboratory (MCWL, Quantico)
to be a tangential analysis of the domain that is regarding unpublished work.
immersive simulation, including application not just as
skills training, but as a method to inoculate against, or [9] The VIRTSIM full-body, real-time team immersion was
treat, stress reactions. The proceedings address addressed in [7]. Additional approaches to body
required attributes of the immersive space, including motion capture and team immersion training exist.
the intelligent adversaries and other VH participants. The two cited here use head tracking and body
position/posture sensors, along with weapon-mounted
On-line proceedings of the 2010 Navy Marine Combat locomotion input devices, to approximate full-body
Operational Stress Control Conference (COSC), San motion input to the avatar. These and other
Diego, CA, 18-20 May, 2010. technologies are addressed in [4], [5], and [6]. In
addition to the instrumentation used to capture motion
http://www.med.navy.mil/sites/nmcsd/nccosc/coscCon in those systems not using natural motion (i.e., all but
ference/Pages/2010/coscConference2010Media.aspx the VIRTSIM device), the algorithms to map the
sensor and input device C2 to the motion of the avatar
can vary, as described by Templeman, et al:
[6] A comprehensive analysis of Live and Virtual simulation for
Infantry small units was conducted in the recent Joint Templeman, J.N., Sibert, L.E., Page, R.C., Denbrook,
Capability Technology Demonstration (JCTD) P.S. (2006) Immersive Simulation to Train Urban
“Future Immersive Training Environment” or FITE. Infantry Combat, in Virtual Media for Military
The FITE final report assessed not only the Live and Applications (pp 23-1 – 23 -16). Meeting Proceedings
Virtual experiments conducted during the JCTD, but RTO-MP-HFM-136, Paper 23, Neuilly-sur-Seine,
also included a survey of available immersion France: RTO.
products and technologies.
[10] Herrero, Pilar, de Antonio, Angelica, A human based
Joint Forces Command, Spiral 1 FOTE JCTD, perception model for cooperative intelligent virtual
“Limited Joint Operational Utility Assessment Report,” agents, Facultad de Informática. Universidad
June 2010. Politécnica de Madrid. Campus de Montegancedo S/N.
28.660 Boadilla del Monte. Madrid. Spain
[7] Although reference to whole body Virtual immersion via
means of networked camera participant digitization is [11] TRADOC Pam 525-8-2, The U.S. Army Learning concept
made in [6], the surveyed technologies did not include for 2015, 20 January 2011.
the new product by Motion Reality, Incorporated
(MRI), VIRTSIM™. Prior camera-based, arbitrary
volume traversal technology was either not real-time
or not multiple participant. VIRTSIM is both. The
technology used by the VIRTSIM device is an
extension of the technology licensed by MRI to Giant
Entertainment, and with which the Academy Award™
for Technical Achievement was won by MRI in 2005
for their work mapping human motions to the Gollum
creature, and with which the Avatar movie generated
the human-based character animations. The
observations made by the author as reported in this
paper are from exercises conducted in the VIRTSIM
device.
http://www.youtube.com/watch?v=UoCtyNb4ESs
[8] The Navy and Marines have conducted research on the
number of controlling humans that are required for the
introduction of intelligent behavior in both VH and
USV robotics. The cited ratios are from conversation
between the author and representatives of the Marine