VIEWS: 14 PAGES: 77 POSTED ON: 7/17/2012
Interaction with sound & music (pt. I) Lalya Gaye, Viktoria Institute Information Art & Design Course Tama Art University, Tokyo, Japan 26 september 2005 Introduction Interaction with sound & music Aims and scope • Introduction to interaction design and prototyping methods for the design of interfaces for: – music performances / installations – use of sound in everyday life • Project examples • Simple prototyping activities Introduction Schedule * Monday 13.00-16.00 • Brief overview of my work • Design space: music controller • Design space: everyday interfacing with sound (e.g. mobile music technology) * Tuesday 13.00-16.00 • Prototyping methods • Case study • Prototyping activities Lalya Gaye Future Applications Lab, Viktoria Institute Göteborg, Sweden Future Applications Lab One group at the research institute in applied information technology Viktoria Institute, in Göteborg, Sweden. 6 PhD students + one senior adviser (Lars Erik Holmquist) Research: Ubiquitous computing, mobile and locative media, information visualisation, tangible computing, human-computer interaction, interaction design, etc www.viktoria.se/~lalya Education BSc Physics – University of Geneva, Switzerland MScEng Engineering Physics – Royal Institute of Technology (KTH), Stockholm, Sweden PhD student in Informatics / Applied Information Technology – University of Göteborg / IT-University of Göteborg, Sweden www.viktoria.se/~lalya Research Focus Ubiquitous Computing for Everyday Aesthetic Practices Keywords: Ubicomp, sound, photography, physical interaction, everyday life Method Experimental projects: - Concepts - Prototypes - User studies Workshops: Mobile music technology Design sketches: Aesthetics of physical interaction www.viktoria.se/~lalya Sonic City www.viktoria.se/fal/projects/soniccity Lalya Gaye (FAL, Viktoria Institute), Ramia Mazé (Re:Form, Interactive Institute), Daniel Skoglund (8Tunnel2), Margot Jacobs (Re:Form, Interactive Institute) Creating a real-time personal soundscape of electronic music by walking through and interacting with urban environments Technology: a wearable using sensor-based information to control real-time audio processing of urban sounds and turn them into music www.viktoria.se/~lalya Tejp www.tii.se/reform/projects/pps/tejp.html www.tii.se/reform/projects/pps/tejp.html Margot Jacobs (Re:Form, Interactive Institute) & Lalya Gaye (FAL, Viktoria Institute) Embodied interaction with locative media (e.g. space annotations) Example: Audio tags on walls whisper to people leaning towards them; space of intimacy in public space www.viktoria.se/~lalya Context Photography www.viktoria.se/fal/projects/photo Maria Håkansson, Sara Ljungblad, Lalya Gaye, Mattias Rost (FAL, Viktoria) Panajotis Mihalatos (Art & Technology, IT-University in Göteborg) A context camera captures the invisible context of a scene with sensors (e.g. level) and translates it visually into the resulting still picture noise, as you are taking it www.viktoria.se/~lalya Tap-n-Bass http://www.timebend.net/projects/?tap_and_bass Lalya Gaye, Alexander Berman, Valerie Bugmann (Goutte d’Or) Live drum-n-bass with re-mixed tap-dance shoes Sound of wired-up tap dance shoes is re-mixed in real time into live drum&bass www.viktoria.se/~lalya Attack of the PANTS (Portable Art Noise Things) David McCallum, Lalya Gaye, Tobias Good (Fringe) DIY workshop for amateurs with simple electronics: Building portable audio units with contact microphones and speakers made out of junk boxes www.viktoria.se/~lalya Mobile Music workshops www.viktoria.se/fal/events/mobilemusic Lars Erik Holmquist (FAL, Viktoria), Atau Tanaka (Sony CSL Paris), Lalya Gaye (FAL, Viktoria) , Frauke Behrendt (Univ. of Sussex), Drew Hemment (Univ. of Salford, Futuresonic + PLAN) Mobile Music: Music technology meets mobile computing. Devices used anywhere, with awareness of place, in distributed / ad hoc networks… Workshop sessions with presentations, in-depth discussions, group brainstormings, hands-on activities… www.viktoria.se/~lalya Interfacing with sound Interfacing with sound Types of interaction with sound * Performance/installations vs everyday use • Music making: performance / composition • Sound-art installations • Sound as design material • Mobile audio • Ambient displays • etc Interfacing with sound Design of music controllers Design: music controllers Acoustic vs. electronic instruments * Acoustic instruments • Sound source and interface are the same artefact • Nature of expected sound & laws of physics dictate the instrument’s form and how it is designed • Fixed and relatively transparent mapping • Acoustic + haptic feedback Design: music controllers Acoustic vs. electronic instruments * Electronic instruments • Sound soure and interface separated mapping, interaction and physical attributes are free • Need for methods and constraints in order to fulfill criteria of expressiveness, transparancy, audio quality • Need for feedback Design: music controllers Issues * Purpose of design Users Experts vs. amateurs? • Uses Composition? Performance? Education? Cognitive stimulation? • Designing an… Interface? Instrument? Composition tool? Design: music controllers Issues * Criterias • Relationship between performer and audience • Physical effort • Complexity / transparency • Ergonomics • Cultural context It is not just about producing sound, it is about the whole experience of producing sound Design: music controllers Interaction loop * Simplified model input mapping output sound interface generator 1rst fb 2nd fb Design: music controllers Interaction loop * Simplified model input mapping output sound interface generator 1rst fb 2nd fb Design: music controllers Interfaces * Object-based Starting with existing instruments - augmented (hyperinstruments…) - digitalised (ex: piano synth) - interface used as controller (ex: MIDI keyboard) Use metaphor of object Taku Lippit, ITP/NYU, 2002-03 Machover & Ma, Hypercello, MIT, 1991 Design: music controllers Interfaces * Object-based Repurposed everyday objects and materials: water, fabric, chemicals, vegetables … Daniel Skoglund, 8Tunnel2 Particles, Horio Kanta, 2003 MIDI Scrapyard Challenge, Brucker-Cohen & Moriwaki, 03-04 Design: music controllers Interfaces * Object-based Take advantage of the material properties of objects f.e.x bendable, conducts electricity, etc Take into consideration human activities surrounding the objects: build upon it and / or break from it Design: music controllers Interfaces * Body-based Human body as start for design The Hands, Waisvicz, STEIM, 1984 Design: music controllers Interfaces * Body-based Human body as start for design: - Ergonomics - Existing gestures - Expressive qualities of human movements - Scale and continuity of movements Design: music controllers Interfaces * Environment-based Interactive environments - Reactive floors - Digital realm: networked audio Everyday environments, etc Magic Carpet, MIT Medialab, 1996 Global String, Tanaka Sonic City, Gaye et al., 02-04 & Toeplitz, 1998 Design: music controllers Interfaces * Environment-based Take advantage of the features of space - Interactive environments: many people together, control of interaction parameters… - Everyday environments: rich environment, unpredictable, dynamic, heterogeneous Design: music controllers Interfaces * Wearables Musical jeans jacket (MIT Medialab, 1992) Tgarden (FoAM & sponge, ~2001) Expressive Footwear (MIT, 1997-2000) ensemble (Kristina Andersen, ~2003) Design: music controllers Interfaces * Wearables Intimate interfaces Body movement and posture Theatrical vs. daily life dimensions Design: music controllers Interfaces * Representations - Tangible algorithms Audiopad, Patten, Medialab, 2001 Block Jam, Newton-Dunn et al., Sony CSL, 2002 - Virtual instruments Mulder, Simon Fraser Univ., 199? - Screen-based (laptop musicians using MAX/MSP, Pd, etc) Design: music controllers Interfaces * Representations Taking familiar sound manipulation metaphor and making it tangible, into space. Design: music controllers Interfaces * Circuit bending Hacking is fun! Modified Toy Orchestra Design: music controllers Interfaces * Hybrids Controlling e.g. audio and visuals together Balancing and adapting interaction so that both dimensions are satisfactory Manual Input Sessions, Golan Levin, 2005 Design: music controllers Interaction loop * Simplified model input mapping output sound interface generator 1rst fb 2nd fb control Design: music controllers Control Levels of indeterminancy • Control vs. randomness ( interactive improvisation) • Total predeterminancy: push a button deterministic output • Total undeterminancy: random machines • Unexpected vs. expected input / output Control characteristics • Continuous vs. discrete control • Implicit vs. explicit • Micro- to macro-level control: sound spectrum to details of articulation to overall structure Design: music controllers Interaction loop * Simplified model input mapping output sound interface generator 1rst fb 2nd fb Design: music controllers Mapping * Issues • Complexity to stimulate creativity • Transparency to keep link between input and resulting sound (otherwise, danger of loosing the audience) Design: music controllers Mapping * One-to-one input mapping output X X X X X X • Each independent input assigned to one musical parameter • Simplest mapping scheme, but usually the least expressive Design: music controllers Mapping * One-to-many input mapping output X X X X X X • One input controls more than one simultaneous musical parameter • Conductor model: provides a macro-level expressivity control, but does not allow access to internal (micro) features Design: music controllers Mapping * Many-to-one input mapping output X X X X X X • Many inputs coupled to produce one musical parameter • Requires previous experience with the system in order to achieve effective control • But far more expressive than the simpler unity mapping Design: music controllers Mapping * Many-to-many input mapping output X X X X X X • Many inputs coupled to many musical parameters • Control on different levels Design: music controllers Interaction loop * Simplified model input mapping output sound interface generator 1rst fb 2nd fb feedback Design: music controllers Feedback Feedback • Helps articulating control • Passive vs. active • From mono- to multi-modal (modalities: audio, haptic, visual) • 1rst FB: from interface • 2nd FB: audio Feedforward • Guides user by providing information about the internal state of the system (as opposed to information from output) Design: music controllers Interaction loop * Simplified model input mapping output sound interface generator 1rst fb 2nd fb Design: music controllers Output * Mechanical Guitarbot (Eric Singer et al., LEMUR, 2003-) * Tactile output (haptics) Cutaneous Grooves (E. Gunther, MIT Medialab, 2001) Design: music controllers Output * Alternative speakers SoundbugTM speakers & piezos Spherical speakers (Curtis Bahn) Flower Speakers (LET’S corporation, Japan, 2004) Design: music controllers Interaction loop * Simplified model input mapping output sound interface generator 1rst fb 2nd fb General forms of interaction Design: music controllers Interaction * User movement - Choreographed body movement - Traditional instrumental gesture - Novel gestures Dark around the Edges, Machover & Yoyo Ma, The Hands, Waisvicz, STEIM, 1984 Winkler, 1997 Hypercello, 1991 Design: music controllers Interaction * User movement - Full-handed gesture Unfoldings, Interactive Stranglophone, Inst., 2003 Sharon, ITP/NYU, 03 - Empty-handed gesture Lady glove, Bongers & Sonami, 1991 + Unvoluntary movements, embodied actions... Design: music controllers Interaction * Real-time music Improvising new music vs. interpreting existing one (conductor model) Radio Baton, Max Mathews, 1987 vs. navigating through non-linear musical narratives Design: music controllers Interaction * During performance Tooka, Fels et. Al, UBC, 01-03 Interaction with environment, audience, etc - Performer-performer - System-audience Crackle-family, STEIM, 1976 - Performer-system-audience Dialtones, Golan Levin, 2001 Design: music controllers Interaction * During performance - Audience as collaborative performer Sine Wave Orchestra, Tokyo, 2003-04 - Private performances in public spaces …or over the internet Le Placard headphones concerts Interfacing with sound: Performance/installations vs everyday use Properties of sound in everyday life • Ubiquitous (sometimes obtrusive) • Dynamic and transient • Broad yet subtle information carrier (emotions, data) • Socio-cultural meaning • Strong link to space and time • Physicality (body and space) • Additive: layers • Foreground vs. background awareness -> implicit vs. explicit interaction Sound in everyday interactions Audio as input Examples from art & research Blendie (Kelly Dobson, MIT Medialab, 2003-04) Context Photography (FAL, Viktoria Institute, 2003-04) -> physicality, cultural meaning… Sound in everyday interactions Outputing sound * Ambient audio displays - Street crossing auditory displays etc - Sonification of network activity: AmbientROOM (Hiroshi Ishii et al., MIT, 1996-97) -> Peripheral awareness Mobile music and locative audio Mobile music and locative audio Locative audio in public space * Motivations Sound as public display Peripheral awareness Community re-appropriation of public space Mobile music and locative audio Locative audio in public space * Space annotation Hear&There (Rozier, MIT Medialab, 1999) Tacticle Sound Garden (Mark Shepard, 2004) Tejp / Audio tags (PLAY & FAL, 2003-04) Mobile music and locative audio Locative audio in public space * Radio pirates Bit Radio (Bureau of Inverse Technology) Key Chain Radio Station (Rikako Sakai, Ivrea, 2004) Mobile music and locative audio Mobile Music Technology Music technology meets mobile computing. Devices used anywhere, with awareness of place, in distributed / ad hoc networks… Device follows user’s displacement and connects to the world (physical, social, located virtual) - Mobile music making, listening, sharing - Wearable audio - Sound walks, etc Mobile music and locative audio Mobile music * Mobile music sharing Social aspect of mobile computing: ad hoc networks, distributed social networks, etc -> spontaneous and situated music sharing with people in public space Mobile music and locative audio Mobile music * Mobile music sharing SoundPryer (Mattias Östergren, Interactive Institute, 2001) TunA (Arianna Bassoli et al., Medialab Europe, 2002) Push!Music (Håkansson et al., Viktoria Institute, 2005) Mobile music and locative audio Mobile music * Mobile music sharing Bass Station (Mark Argo & Ahmi Wolf, ITP/NYU, 2003) Mobile music and locative audio Mobile music * Mobile music making Music making away from computer screen or performance setting: in the everyday Sensor technology + GPS -> situated music making Ad hoc & distributed networks throughout the city -> collaborative music making etc Mobile music and locative audio Mobile music * Mobile music making Sonic City (Gaye et al., FAL & PLAY, 2002-04) Malleable Mobile Music (Atau Tanaka, Sony CSL, 2004) Mobile music and locative audio Mobile music * Mobile music making Sound Mapping (Mott et al., Reverberant, 1998) Sonic Interface (Akitsugu Maebayashi, 1999) Mobile music and locative audio Mobile music * Mobile music making CosTune (Nishimoto, ATR, 2001) Sound Lens (Toshio Iwai, 200?) Mobile and locative sound ”Walking through sound” (D. Toop) * Sound-art installations Electric walks (Christina Kubisch) Drift (Rueb) * Walking through digital space Seven Mile Boots (Beloff et al., 2003-04) * Non-linear audio narratives The Case at Kulturhuset (Knifeandfork, 2004) Mobile and locative sound Wearable audio ”Personal instruments” (Krzysztof Wodiczko, 1969) (Chelle Hugues, RCA/CRD, 2000) Mobile and locative sound Wearable audio Nomadic Radio (Nitin Shawney, MIT Medialab, 1998) Sonic Fabric (Alice Santaro, 2002) Design: everyday interactions Considerations • Is the interaction implicit or explicit? Is the awareness foreground or background? • Mapping issues too • How does the sound fit in the environment? • Cultural assumptions about how things sound (familiar movie effects vs. unusual real life sounds) • Emotional content and conotations • Role in everyday life (balancing everyday activities with interaction with sound, etc) Etc. Design: everyday interactions Considerations How to find out: interaction design methods & prototyping! For tomorrow Design exercices • Get into groups of 4 people • Discuss potential sound application ideas (very basic and simple ones!) • Choose interesting everyday artefacts to bring tomorrow • Tomorrow: design exercises! ありがとうございました！ またあした !
Pages to are hidden for
"PowerPoint-presentation - Viktoria Institute"Please download to view full document