Sensory Puzzles

Document Sample
Sensory Puzzles Powered By Docstoc
					                                              Sensory Puzzles
   Tamara M. Lackner        Kelly Dobson           Roy Rodenstein                                 Luke Weisman
     MIT Media Lab      MIT Visual Arts Program    MIT Media Lab                                    MIT AI Lab
      20 Ames Street    265 Massachusetts Ave.      20 Ames Street                             545 Technology Square
        E15-020B               N51-315                 E15-450                                        E43-827
  Cambridge, MA 02139 Cambridge, MA 02139 Cambridge, MA 02139                                  Cambridge, MA 02139
     +1 617 253 1401       +1 617 253 5229         +1 617 253 0323                                +1 617 258 7470                    

ABSTRACT                                                         more engaging. Our puzzle's use of audible cues and
Traditional jigsaw puzzles provide interesting visual            tangible acoustic topologies on the pieces broadens the
stimulation, but hold diminished rewards for visually            sensory dimensionality of the interaction. Beyond being
impaired users. We have built a puzzle whose pieces play         more engaging, the multi-modal experience of the Sensory
portions of music and include a tangible topology                Puzzle can be a vehicle to teach children and adults about
representing their contents, giving auditory and tactile         music theory and to encourage people with differing
feedback about the pieces’ role in the puzzle's overall          sensory abilities to play with the puzzle together.
solution. In addition to providing entertainment, the
                                                                 RELATED WORK
puzzle    pieces   support    learning   about    music,
experimentation, and social, collaborative construction.         The American Toy Institute, Inc. and the American
By employing multiple senses — hearing, touch, vision—           Foundation for the Blind [1] publish an annual guide
our sensory puzzle provides a rich interaction, play and         containing considerations for and examples of toys for
learning space accessible to individuals with various            visually impaired children, although the primary focus of
sensory abilities.                                               most toys listed continues to be sighted users. Our work is
                                                                 more explicit about designing for differing sensory
Keywords                                                         abilities, as in [2].
Puzzles, multimodal, tangible interfaces, collaboration,
                                                                 The Brain Opera project [3] includes rubber-cast shapes
entertainment, music, education.
                                                                 attached to surfaces which users can push on to generate
INTRODUCTION                                                     sounds, although our Sensory Puzzle is more directly
Sensory experiences, whether labeled as play in children or      related to previous work in tangible soundscapes [5] and
as cultural experiences in adults, are the means by which        music visualization [6].
people process what they take in from the world in order to
                                                                 THE PUZZLE
learn about and develop themselves and their relationship
                                                                 The puzzle is made up of pieces in the form of rectangular
to their environment. Sight is often privileged over other
                                                                 blocks. Each piece contains a small hardware board that
senses in society, and concomitantly in the design of
                                                                 holds a portion of the song the puzzle represents, and can
games and learning tools.        We hope to encourage
                                                                 play this portion through a speaker. Additionally, each
heightened awareness of the value and pleasure of other
                                                                 piece is topped by a physical topology corresponding to the
senses through our multi-modal puzzle. Our Sensory
                                                                 pitch and duration of its portion’ musical notes as well as
Puzzle, inspired by a childhood love of jigsaw puzzles and
                                                                 its tempo. The hills and valleys of the topology are good
the desire to share that feeling with visually impaired
                                                                 visual and tactile descriptors of the portion of music
children, provides particularly rich entertainment and
                                                                 contained in the piece (see Figure 1). As with the Tangible
learning possibilities by employing primarily the senses of
                                                                 Audio Landscapes [2], the puzzle provides a metaphor for
hearing and touch.
                                                                 perceiving and operating on complex sound data via
The concept of a Sensory Puzzle as a tangible interface is       physical manipulation and construction. The completed
interesting in several respects. Primarily, it allows for rich   spatial positioning of the puzzle pieces creates a cohesive,
multi-sensory experiences beyond those found in a                structured display of the music's physical soundscape.
traditional, flat jigsaw puzzle. Although visual sensations      With repeated use, patterns of notes may be learned and
are interesting, the senses of touch and of hearing are often    associated with certain pitch, duration and tempo maps
                                                                 [6], thus linking aural experience with visual and tactile
                                                                 interaction. In this sense the puzzle acts as an educational
                                                                 tool. Additionally, the puzzle can serve as an
                                                                 experimentation and synthesis tool, as pieces can be
combined in any fashion to create new sound patterns and        technologies allows each puzzle piece to stand alone,
musical compositions.                                           rather than being tethered to a computer or power source.
The current version of the sensory puzzle uses a “phrase,”      With the sound localized in each piece, the connection
an apparent unitary segment of a song, for each piece. The                           s
                                                                between each piece’ topology and its audio is much more
length of each piece’ contents is an interesting variable,
                       s                                        perceptually effective than sound output from a single
relevant to users of different ages and musical skill levels    source, which would destroy the holistic nature of
as well as to those interested primarily in entertainment or    the pieces. In addition, stand-alone pieces are imperative
in composition. The first song we have built pieces for is      for free manipulation and interaction, and for maintaining
Mozart’ Eine Kleine Nachtmusik, with phrases usually
         s                                                      the emerging, connected quality of jigsaw puzzles.
around nine notes long. Our prototype consists of four
puzzle pieces; putting pieces in close proximity results in
the pieces sequentially playing their respective portion of
music. If the puzzle pieces are put together incorrectly, the
resulting combination of music will of course not sound
like the target song. Nevertheless, pieces which don't “fit”
together can still be associated with each other to create
new patterns and combinations of sounds. The puzzle,
therefore, acts simultaneously as a game, a creative toy,
and an educational tool. When successfully ordered, the
four pieces play the first few bars of the Mozart song.                Figure 2: Components of a puzzle piece—
Currently the puzzle aspect requires placing the pieces in            Crickets, MIDI board, amplifier, and speaker
essentially linear fashion, with time the guiding axis and
                                                                FUTURE DIRECTIONS
sequential order the main variable. We are still exploring
                                                                We would like to push the tangible aspect of the puzzle
interesting ways of mapping time and music to the puzzle
                                                                pieces and allow users to pinch a node in the topology to
                                                                have it vibrate with the sound of that note, increasing the
                                                                pieces’ usefulness for deaf users. In terms of musical
                                                                construction, orchestral songs might be broken up into
                                                                component instrument to explore harmony. Interestingly,
                                                                concepts from artificial life might be applied to pieces so
                                                                that they follow simple rules such as playing at a constant
                                                                tempo but interact with neighboring pieces to determine
                                                                which note they play at any given time.
  Figure 1: Two puzzle pieces with their corresponding
                                                                1. American Toy Institute, Inc., American Foundation for
                                                                   the Blind, Guide to Toys for Children Who are Blind or
TECHNOLOGY AND MUSICAL CONTENT                                     Visually Impaired, 1998-1999,
Our prototype of the Sensory puzzle was made using the          2. Tim Anderson, Clare Smith, “Composability”:
programmable bricks technology developed by members of             Widening Participation in Music Making for People
the Epistemology and Learning Group at the MIT Media               with Disabilities via Music Software and Controller
Lab [4]. The base of each puzzle piece is comprised of two         Solutions, Proceedings of the Second Annual ACM
programmable bricks (see Figure 2), called Crickets, each          Conference on Assistive Technologies, 1996, p. 110
with an infrared transmitter and receiver on one side. The
Crickets are wired together using motor outputs and sensor      3. Maribeth J. Back, Sound design for Brain Opera's
inputs and face opposite directions, which allows bi-              Mind Forest: Audio for a Complex Interactive System,
directional communication between puzzle pieces so that            Proceedings of the conference on Designing Interactive
they can be placed in any sequential order. This bi-               Systems, 1997, pp. 23–25
directional communication mechanism is also used with a         4. Mitchel Resnick et al, Digital Manipulatives: New Toys
simple protocol to have pieces play in sequence; thus when                                             98,
                                                                   to Think With, Proceedings of CHI ’ Pages 281-287.
multiple pieces are placed close together they play one         5. Nick         Sawhney,     Tangible       Soundscapes,
after another in the order in which they are arranged,   
ensuring a continuous and coherent rendering of the song.          gible/soundscapes.html
In order to achieve high-quality sound, the Crickets write      6. Sean M. Smith, Glen N. Williams, A Visualization of
via their bus to a MIDI board, which is then connected to          Music, Proceedings of the conference on Visualization
an amplifier and speaker. The combination of these                 '97, 1997, pp. 499-503