Document Sample
					                       THE USE OF INTERACTIVE VIDEO
                                 ref: 199201

                                         Jim Schofield
                              Director of Information Technology
                           Goldsmiths' College, University of London
                                New Cross, London SE14 6NW
                                  Tel: 081 692 7171 ext 2035


                                    Ghaffar Pourazar
                         Performing Artist & Computer Animator,
                                    Visiting Lecturer at
              Bournmouth University, Bournmouth College of Art & Design and
                     Performing Arts & Technology School, Croydon

                                   75 Swanbourne, Wansey St.,
                                        London SE17 1UT
                                     Tel & Fax: 071 703 5480


The paper describes the existing work of the authors in Interactive Video in the Teaching of
Dance, and in Computer Animation Choreography, then discusses their current work into the
development of a "parametered 3D man", which can be used for both the digitisation of dance
movements from video, and the demonstration of original choreography by its animated
movement. The derivation of "real movement" constraints from the digitised time dependant
databases, using artificial intelligence techniques, is explained, and the application of these to the
driving of the figure in "real movement" animation will be explained. The distilling from actual
movement videos of basic movement primitives and their use to build movement phrases,
sentences and paragraphs will be dealt with, as will the relationship of such work to the currently
available forms of dance notation.

Key Words: interactive video, computer animation, dance notation, artificial intelligence,
teaching of dance, choreography.
Previous Work                                         reference image, though reduced in size in one
                                                      corner of the screen. The access to these images
Jim Schofield specialises in the use of               was controlled by software written initially for
computers in control, and has worked for 5 years      the BBC B microcomputer and later for the IBM
in the area of Interactive Video in the teaching of   PC microcomputer, which enabled instantaneous
Dance. His work with Jacquie Smith of Bedford         switching between views and close-ups, allowed
College of Higher Education led to the release in     slow motion, the addition of an extra sound track
1989 of the Dance Disc, a Laser disc of dance         and the simultaneous display of synchronised
performance for one of the studies in the 1989        dance notation alongside the moving image of
GCSE dance examination used in schools across         the dance itself. Various graphic overlays were
England and Wales. This disc won an award at          generated by the computer to facilitate the
the B.I.V.A. in 1989.                                 accessing and interpretation of the images. One
                                                      such was a small map of the dance space where
Ghaffar Pourazar has spent the last 8 years           an animated dot, in synch with the video action
training working and teaching, equally, in two        on screen, indicated the current position of the
areas of Mime and computer animation. His             dancer. Other overlays gave an indication of the
focus has always been on the dynamics of human        time elapsed from the beginning of the dance, the
movement and his projects have often combined         bar count, and several indicators of the current
 Animation, Mime and Performing Arts in               view options available and their modes of use.
exciting and innovative ways. His last computer       Apart from the detailed optional views there
Animation piece (Mr Bojangles) combined               were complete full screen sequences of the dance
Traditional & Computer Animation through              used in different ways at different times. The
dancing characters. It was a finalist in two          "artistic expression" mode was a front (audience-
festivals and has been screened in various Film       view) sequence including access to all the detail
& Computer animation festivals in Europe. His         special views and close-up as options. The
last performance (Azarbayjan) brings        live      "learning" mode was a rear sequence with
dancers and performers into an animated               options for dance notation, pause, slow motion,
environment, allowing them to interact with           and reversal of direction.
animated characters on the screen. This work          Access to any part of the dance
has been commissioned for the Mime Festival in        in either mode was possible via
                                                      a       dance       phrase      menu,        and
London, January 93. He hopes to bring down the        restarts and mode changes were
 barrier between the Arts and Technology by           each possible via the touching
speaking both languages fluently and                  of a single button.
communicating between the two groups.

The Dance Disc                                        Mr. Bojangles

The Dance Disc was a series of videoed                For Mr. Bojangles a technique was developed to
sequences of the dance study, taken from              capture and animate the dynamics of human
different angles and orientations, or showing         movement. The focus was the reproduction of
amplified details of particular views.                precise movement dynamics (weight, velocity,
Whenever detailed views were shown, the full          balance) and applying it to animated characters,
figure view was also included as a synchronised       3D and 2D. The capture of the movement was
achieved through a rotoscoping technique where       sequence synchronisation frames and even the
the body was divided into 9 limbs and 13 joints      sets of coordinates that define animation overlays
and the movements were recorded via two              and their synchronisation to the underlying
cameras positioned at 90 degrees from each           video. All these were database sets were built
other. Thus 3D positions for the limbs and the       while accessing and manipulating the actual
joints were captured to give 3D paths in space.      video images on the computer screen. This editor
These 3D paths were used as routes, on which         also reveals inconsistencies in the starts, finishes
the animated figures were driven. Amongst the        etc. of directly parallel sequences, which can be
movements that were captured, acrobatic,             shown both graphically and numerically, and to
walking, dancing and jumping jigs formed the         resolve these errors by adjustments in sequence
main sequences of the piece. The addition of a       boundaries.
traditionally animated figure brought a new
dimension of both fluidity and charm to Mr           Beyond Mr. Bojangles
                                                     Developments to Mr Bojangles have been a
Beyond the Dance Disc                                constant feature since the outset. and have been
                                                     made on both technical and
The experience of seeing the Dance Disc used in      artistic fronts.
the teaching of dance performance not only           Using spline techniques it was possible to divert
showed the success of this method of                 the paths and create new paths, and therefore
communicating the richness and subtlety of the       new movements and new dynamics. Some of the
dance, but also led to definitions of                movements looked realistic and some looked
improvements and developments of the                 stylised, wobbly, floaty, stabby, light, heavy etc...
techniques involved. The control keys were           This added a new 'feel' or attitude to the animated
developed into more intuitively used interfaces      figure. Next, using differently shaped and
by the use of toggle functions (to allow looping     coloured objects for the limbs, it was possible to
round a movement with a single finger) and           make other physical, cultural and even emotional
combined single shot/slow motion keys, to give       distinctions between the figures i.e. fat, female,
greater, and dynamic control in complex              old, Italian, African, dancer, sportsman, childish,
movements. In addition, the same sets of             etc... Though in all these cases, the animated
functions were mapped onto mouse buttons,            figures were making exactly the same
which became active in these functions when the      movement. Finally the permutations between
mouse was pointed at appropriate function            varying splines for paths and varying shapes and
buttons on screen. The Time thermometer (time        colours for the limbs gave us a powerful tool for
elapsed) overlay was developed into a mouse          creating figures which could dance and perform
controlled menu for instant access to any part of    and give more than physical comments with their
the dance, and visual rhythm indicators were         body. This was the beginning of choreographic
designed to help cue the start or turning point of   developments in the research.
a movement. The tracing of trajectories of given
points on the body facilitated the teaching of the
"feel" of certain movements and the adjustments      Azarbayjan
necessary in mid movement.
                                                     The research in human dynamics and animation
The Video Editor                                     (artistic or scientific) continued     into a
                                                     performance project called AZARBAYJAN.
Housekeeping overheads led           to further      Here the performer interacts with an animated
developments such as the writing     of a Video      environment and dances with animated
Editor. This allowed databases to    be built of     characters, in order to create stunning visual
starting and ending frames of         sequences,     performances. The work with Mr Bojangles not
only produced some of the sequences in               would be limited by constraints derived from
AZARBAYJAN, it also laid a basis for the visual      real movements, ideally from the actual dancer
experiments with dynamics, shapes and colours,       who is now to be overlaid with animated graphic
the charm and aesthetics of the animations to be     alternatives.
exposed to audiences, on big screen.
                                                     The Parametered Man (II)
The Parametered Man (I)
                                                     The 9 limbs and 13 joints in the figures of Mr
Further developments of the ideas inherent in the    Bojangles formed the starting point for the
Dance Disc revolved around the problem of the        figures. The objective was simplicity and
need to demonstrate alternatives to any particular   economy in the number of variables used. The
movement available as part of the dance video on     trunk was missing, yet even subtle movements
disc.                                                came through to the audience. A simple and
In using the techniques of the Dance Disc in         economical approach pays off very well in the
choreography exercises, the number of                long run, and will stay our approach to the
alternatives to a movement might be quite large,     parametered man. However,
and for this to be captured for use in multiple      there are new factors to be addressed, in that we
video sequences would not only overburden the        will be able to extract the properties of the
capacity of the disc, but would likely be an         original
intolerable strain on the performers to capture      dancer/actor, either as constraints for the style,
every nuance correctly.                              tradition or type of the dance, or as those for the
A possible alternative approach would be to have     dancer.
a parameter driven animated wireframe overlay        Let us consider the individual traits of the
figure (The Parametered Man) which could be          actor/dancer which should be deliverable to our
shown in a series of different super-imposed         Parametered Man. Lets say that our dancer from
variations on the basic movement in the              China is an expert in Peking Opera, he is forty
underlying video.                                    years old, and he is short and carries a slight
These would be held as complete parameter            injury on his left shoulder. Now you understand
sequences which could drive the figure for each      what kind of considerations we must give to the
alternative movement, and would demand very          parametered man.
much less in storage requirements.                   Performers of different sizes and abilities, all
The "solution" has, of course, a set of new          around the world, will have learnt highly stylised
problems associated with it. The most                movements which have traditional meanings and
fundamental is that a simple animated                their own inbuilt constraints. We must be able to
parametered graphical figure would be                extract these natural constraints, and reapply to
overlaying crude artificial movement over real       driven movement by our parametered man. In
videoed movement. This is because "any               addition, we should be able to apply our own
movement or speed of motion would be                 "created" constraints to movements, thus
possible", and the alternative movement as           choreographing entirely new movements.
presented by the parametered figure might
therefore not be interpretable in terms of the       The Functionality of the Parametered Man
original video movement.
For a parametered figure to be successful it         The Parametered Man is an articulated 3D
would have to move naturally, or put another         graphic figure, with a hierarchy of hinged units,
way, it would have to move under the same            flexible enough to allow the matching of the
constraints as did the original videoed              figure to a specially shot underlying video of
movement.                                            dance, or to be used to demonstrate movement of
The problem thus becomes the development of a        itself as an animated sequence.
parametered driveable figure, whose movements
                                                      movements, so that different constraints will be
Digitising Dance Video                                applied near to zero relative velocities.

To capture three dimensional dance movement
from video, more is required than one overlay
figure matched to a single viewpoint sequence. It     Constraints & the Diverse Primitives of
will be necessary to employ two linked figures        Movement
viewed from two perpendicular standpoints, each
being matched to a corresponding underlying           In Kabuki the female role, which is often played
video view.                                           by males, has a precise and distinct walk with it's
In addition, it will include exterior graphical       own constraints and freedom of movement
prompts, suitably geared, to facilitate feedback in   dynamics, determined by a style and a traditional
placing the figure with precision.                    code of conduct. It is a highly stylised form of
The problems of the usual parent-child                theatre and the rules apply to each character i.e.
relationships between elements, when dealing          man, God, etc.
with real movements which have to be sourced          These rules provide the initial properties for our
from changing "roots", will also have to be           movements of the kabuki dancer.
overcome.                                             Not only the dynamics of the dancer are captured
The positions and orientations of all components      but certain properties are extracted, which act as
of the figure for each frame in the movement          intelligent constraints.
being digitised will be stored as time dependant      These inherent rules and constraints apply to all
sequences, which can be processed to define           forms of movement & dance; some very tightly
inherent constraints on the movement, or used         defined, such as ballet, and others not so, i.e.
directly to drive the parametered man in              contemporary dance.
"following" the captured movement.                    Once the constraints are determined, they can act
                                                      as limiters to reveal wrong movements, or tools
                                                      to choreograph new ones.
Deriving Movement Constraints
                                                      Dancer, Style & Cultural Factors
To enable the parametered man to be moved in
new ways, yet maintain "natural movement" it          By capturing movements of dances such as :
must be constrained by ranges and limits on its       Katakali, flamenco, lezgy, tango, samba
parameters derived from previously digitised          and the many varieties of dance and movement
movements.                                            rituals all around the world, we create a library of
Two main kind of constraints can be distilled         movements. The properties and constraints
from the database of a given dance or sequence        which we can extract from these movements
of movements.                                         form a powerful tool for the choreographer and
The first are "space" constraints which define the    the dancer.
limits of orientation or position and could be        This applies equally to different forms of
extracted as maximum and minimum values.              traditional theatre which are highly physical, i.e.
Secondly, there are "dynamic" constraints which       Commadia Del'Arte, Noa, Peking Opera,
are to do with velocities and accelerations. These    Kabuki, most masked forms of theatre,
will be extracted by calculating running averages     pantomime, etc etc
over the most recent series of frames as the
database is be traversed, and again processed to      Driving The Parametered Man
deliver maxima and minima over the whole
piece.                                                Once a set of constraints has been derived from
It is clear that the dynamic constraints will have    previous movement, it will be possible to drive
different properties at the start and finish of       the figure in new ways consistent with these
limitations.                                           choreographers will appreciate seeing their
As the figure is moved into each new position,         notation applied to a moving parametered
any attempted transgression of a space constraint      man/woman, and will welcome the chance of
will be thwarted by the driving software which         demonstrating, tweaking and correcting their
will immediately indicate the error, and will          notation and even creating and adding new
show as an alternative the actual limiting position    movements.
in the given direction of movement. For dynamic
constraints, the driving software will be taking
running averages, (as when deriving constraints)
so that velocities and accelerations are compared
with the stored equivalents. Once again, any
transgression of the constraints will immediately
be signalled and the limiting position, (to stay
"legal") will be offered as an option. It will not
be necessary to only drive the figure by
sequential frame by frame movement. As long as
the actual number of frames between separated
key positions is clear, the constraining software
will be able to make temporary extrapolations,
which can be supplemented by detailed frame-to-
frame check on infill of the missing frames later.
In certain cases it will be possible to generate the
intervening frames, between manually entered
key frames, automatically consistent with the
governing constraints.

The Display of Notated Movement by the
Parametered Man

Already there exist different forms of movement
and dance notations, and almost every culture
has it's own way of recording dance. Often the
notation suits best the dance and style of the
culture it was devised in, and no matter how
universal, it is hard pressed when notating dances
other cultures. After all, which universal dance
notation can cover equally well the
mimical/acrobatic precision of a Peking Opera
performer, the curvy eroticism of Arabic /
Persian dancing, the mathematical rules of
Morris dancing, the physical comedy
combination of Commadia Del'Arte, the strength
and the softness of the Kabuki performer, the
power and the earthy joy of the African dancer,
the ballet lines, Samba, Flamenco, Tango, and
hundreds of others?
Even the most serious of the dance notators,

Shared By: