Docstoc

dynamic_multimedia_mum2003_talk

Document Sample
dynamic_multimedia_mum2003_talk Powered By Docstoc
					     Dynamic Distributed Multimedia:
Seamless Sharing and Reconfiguration of Multimedia
                   Flow Graphs
        Marco Lohse, Michael Repplinger, Philipp Slusallek

      Computer Graphics Lab, Saarland University, Germany




                                    MUM2003 - Dynamic Distributed Multimedia – p.1/21
                                                     Motivation
•   Increased number of networked mobile and stationary
    multimedia devices
      • Web pads, PDAs, mobile phones, PCs

               Mobile system           Stationary system




               Limited processing      Processing power
                power
                                       I/O capabilities
               I/O capabilities          Large screen
                 Small screen            High fidelity audio
                 Mono audio              DVD drive
                                         TV receiver


•   Ubiquitous multimedia?
                                    MUM2003 - Dynamic Distributed Multimedia – p.2/21
                      Dynamic Reconfiguration
•   User and device mobility
•   Use capabilities of surrounding environment
•   Play out of multimedia data during reconfiguration?
•   Seamless and synchronized handover?

      Mobile system            Stationary system

      MP3
      playback




                               MUM2003 - Dynamic Distributed Multimedia – p.3/21
                      Dynamic Reconfiguration
•   User and device mobility
•   Use capabilities of surrounding environment
•   Play out of multimedia data during reconfiguration?
•   Seamless and synchronized handover?

      Mobile system            Stationary system

                                MP3
                                playback




                               MUM2003 - Dynamic Distributed Multimedia – p.3/21
                          Collaborative Scenarios
•   Number of users simultaneously enjoys the same (or
    similar) content using different devices
•   Multiple access of hardware devices?
•   Increase of processing and networking requirements?
•   Synchronized playback on different devices?

        Mobile system 1       Stationary system




                                  DVD playback


                               MUM2003 - Dynamic Distributed Multimedia – p.4/21
                          Collaborative Scenarios
•   Number of users simultaneously enjoys the same (or
    similar) content using different devices
•   Multiple access of hardware devices?
•   Increase of processing and networking requirements?
•   Synchronized playback on different devices?

        Mobile system 1       Stationary system
         audio
         track 0                  video




                                  DVD playback


                               MUM2003 - Dynamic Distributed Multimedia – p.4/21
                           Collaborative Scenarios
•   Number of users simultaneously enjoys the same (or
    similar) content using different devices
•   Multiple access of hardware devices?
•   Increase of processing and networking requirements?
•   Synchronized playback on different devices?

         Mobile system 1       Stationary system
         audio
         track 0                   video



         Mobile system 2
         audio                     DVD playback
         track 1
                               MUM2003 - Dynamic Distributed Multimedia – p.4/21
                                                    Overview
•   Motivation
•   Network-Integrated Multimedia Middleware (NMM)
     • Distributed synchronization
     • Registry service

•   Session sharing
     • Maps new applications to already running
       applications
•   Session handover
     • Migrates parts of a running application

•   Conclusions and future work

                               MUM2003 - Dynamic Distributed Multimedia – p.5/21
                                     NMM Flow Graphs
•   Nodes as smallest processing unit
•   Jacks to connect nodes
•   Jack tags to identify inputs and outputs
•   Formats to type connections
•   Interfaces to access objects
•   Multimedia data buffer and events
                             Application
       INode                     INode                            INode

               "default" "default"   "default" "default"
                           Audio
     Readfile Output Input Decode Output Input Playback
       Node    Jack Jack           Jack Jack     Node
               MP3    MP3   Node   RAW    RAW

                                         MUM2003 - Dynamic Distributed Multimedia – p.6/21
                       Distributed Synchronization
•   Strict separation of intra-stream and inter-stream
    synchronization
     • Locally running controllers
     • Synchronizer adjusts latencies


        Node           InputJack
                                       SyncSinkNode
                   T    T   T
    B          T   B   B    B      B      T

                                        SinkController



                                                                      Synchronizer
        Node           InputJack       SyncSinkNode
                   T    T   T
    B          T   B   B    B      B     T

                                        SinkController

                                                              Network/address space

                                              MUM2003 - Dynamic Distributed Multimedia – p.7/21
                                              Registry Service
•   Administrates locally available nodes and active flow
    graphs (sessions)
•   Graph descriptions as queries
     • Sharing policy for each node

•   Peer-to-peer approach for distributed flow graphs

                                        Audio
                Name Readfile          Decode         Playback
                       Node             Node            Node
                 Type    SOURCE       CONVERTER           SINK

             Interfaces IFileSource       -                -
             I/O format      -        MP3/RAW            RAW
         Sharing policy   Shared      Shared         Exclusive
               Location   Host 2       Host 3           Host 3


                                        MUM2003 - Dynamic Distributed Multimedia – p.8/21
                                                          Registry Service
   •   Administrates locally available nodes and active flow
       graphs (sessions)
   •   Graph descriptions as queries
        • Sharing policy for each node

   •   Peer-to-peer approach for distributed flow graphs


                               Audio
       Name Readfile          Decode      Playback
                 Node          Node         Node
        Type                                SINK
                                                                                Network
                SOURCE       CONVERTER
                                                        Registry
    Interfaces IFileSource      -            -
   I/O format      -         MP3/RAW        RAW
Sharing policy   Shared      Shared      Exclusive
                                                                         Registry          Registry
      Location   Host 2       Host 3      Host 3
                                                               Host 1           Host 2           Host


                                                     MUM2003 - Dynamic Distributed Multimedia – p.8/21
                                     Collaborative Scenarios
•   Shared access to DVD, different audio tracks for
    mobile devices (e.g. iPAQ H3870 with Linux)
     • Shared and exclusive nodes

    Running session 1
                                       Video
                                       Decode        Display
                                        Node          Node
                           "video"
                                                                  Synchronizer
                        MPEG "audio0" Audio
       DVDRead          Demux                       Playback
        Node                         Decode
                        Node           Node           Node

    Query
                        MPEG "audio1" Audio
      DVDRead           Demux                      Playback
       Node                           Decode
                        Node           Node          Node
                                      localhost    localhost




                                           MUM2003 - Dynamic Distributed Multimedia – p.9/21
                                           Session Sharing
 •   Running sessions R1 , . . . , Rn , query Q
 •   Idea
       • Start with node in Q with no incoming edges (e.g.
         source)
       • Try to find ’matching’ node in Ri
       • Try to recursively expand overlapping sub-graphs
          • Re-use overlapped sub-graphs
       • Individual test for edges er and eq

                r      r                             q      q
             tagout tagin                         tagout tagin
er    nr
       out                  nr
                             in   eq      nq
                                           out                          nq
                                                                         in
                fr
                 conn                                   fq
                                                         conn
     s(n r )
         out
                           r
                        s(nin)         s(n q )
                                           out
                                                                        q
                                                                     s(nin)
                                   MUM2003 - Dynamic Distributed Multimedia – p.10/21
                                                 Session Sharing
•   Shared access to DVD, different audio tracks for
    mobile devices (e.g. iPAQ H3870 with Linux)
     • Shared and exclusive nodes

    Running session 1
                                      Video
                                      Decode        Display
                                       Node          Node
                           "video"
                                                                 Synchronizer
                        MPEG "audio0" Audio
       DVDRead          Demux                      Playback
        Node                         Decode
                        Node           Node          Node

    Query
                        MPEG "audio1" Audio
      DVDRead           Demux                     Playback
       Node                           Decode
                        Node           Node         Node
                                     localhost    localhost




                                         MUM2003 - Dynamic Distributed Multimedia – p.11/21
                                                   Session Sharing
•   Shared access to DVD, different audio tracks for
    mobile devices (e.g. iPAQ H3870 with Linux)
     • Shared and exclusive nodes

     Running session 1
                                       Video
                                       Decode         Display
                                        Node           Node
                             "video"
                                                                   Synchronizer
                         MPEG "audio0" Audio
                          MPEG
       DVDRead
       DVDRead           Demux                       Playback
        Node             Demux        Decode
         Node            Node           Node           Node
                          Node

    Query
                         MPEG "audio1" Audio
                         Demux         Decode       Playback
                         Node           Node          Node
                                       localhost    localhost




                                           MUM2003 - Dynamic Distributed Multimedia – p.11/21
                                                 Session Sharing
•   Shared access to DVD, different audio tracks for
    mobile devices (e.g. iPAQ H3870 with Linux)
     • Shared and exclusive nodes

    Running session 1
                                     Video
                                     Decode         Display
                           "video"    Node           Node
                                                                 Synchronizer
                        MPEG "audio0" Audio
                         MPEG
       DVDRead
       DVDRead          Demux                      Playback
        Node            Demux        Decode
         Node           Node           Node          Node
                         Node

    Query
                          "audio1"    Audio
                                     Decode       Playback
                                      Node          Node
                                     localhost    localhost




                                         MUM2003 - Dynamic Distributed Multimedia – p.11/21
                                                 Session Sharing
•   Shared access to DVD, different audio tracks for
    mobile devices (e.g. iPAQ H3870 with Linux)
     • Shared and exclusive nodes

    Running session 1
                                      Video
                                     Decode         Display
                           "video"    Node            Node
                                                                 Synchronizer
                        MPEG "audio0" Audio
                         MPEG
       DVDRead
       DVDRead          Demux                      Playback
        Node            Demux        Decode
         Node           Node           Node          Node
                         Node

    Running session 2
                          "audio1"    Audio
                                     Decode       Playback
                                      Node          Node
                                     localhost    localhost




                                         MUM2003 - Dynamic Distributed Multimedia – p.11/21
                                                  Session Sharing
•   Shared access to DVD, same audio tracks for mobile
    devices (e.g. iPAQ H3870 with Linux)
     • Shared and exclusive nodes

    Running session 1
                                      Video
                                      Decode         Display
                            "video"    Node           Node
                                                                  Synchronizer
                        MPEG "audio0" Audio
       DVDRead          Demux                       Playback
        Node                         Decode
                        Node           Node           Node

    Query
                        MPEG "audio0" Audio
      DVDRead           Demux                      Playback
       Node                           Decode
                        Node           Node          Node
                                      localhost    localhost




                                          MUM2003 - Dynamic Distributed Multimedia – p.12/21
                                                 Session Sharing
•   Shared access to DVD, same audio tracks for mobile
    devices (e.g. iPAQ H3870 with Linux)
     • Shared and exclusive nodes

    Running session 1
                                      Video
                                      Decode        Display
                                       Node          Node
                           "video"
                                                                 Synchronizer
                        MPEG "audio0" Audio
                         MPEG
       DVDRead
       DVDRead          Demux                      Playback
        Node            Demux        Decode
         Node           Node           Node          Node
                         Node

    Query
                        MPEG "audio0" Audio
                        Demux         Decode      Playback
                        Node           Node         Node
                                     localhost    localhost




                                         MUM2003 - Dynamic Distributed Multimedia – p.12/21
                                                      Session Sharing
•   Shared access to DVD, same audio tracks for mobile
    devices (e.g. iPAQ H3870 with Linux)
     • Shared and exclusive nodes

    Running session 1
                                         Video
                                         Decode         Display
                                          Node            Node
                          "video"
                                                                      Synchronizer
                        MPEG    "audio0"
       DVDRead           MPEG   jack
                                            Audio
       DVDRead          Demux              Decode       Playback
         Node           Demux   group
                                                          Node
         Node           Node
                         Node                Node

    Running session 2                   "audio0"
                                         Audio
                                         Decode        Playback
                                          Node           Node
                                        localhost      localhost




                                              MUM2003 - Dynamic Distributed Multimedia – p.12/21
                      Session Sharing Algorithm
•   Branch-and-cut algorithm (depth-first search)
     • Run-time O((branchingf actor)depth )
     • Number of recursions is very small
       • Strict criteria, e.g. the jack tag is unique and has
         to match
       • Cutting of search space
     • Runtime around 100 ms for typical setups

•   Value different overlaps
     • Value function counts overlapped nodes and edges

    ⇒ Multiple access of single hardware device
    ⇒ Reduces processing and networking requirements

                                MUM2003 - Dynamic Distributed Multimedia – p.13/21
                      Dynamic Reconfiguration
•   Playback session running on mobile system
•   Handed over to nearby stationary system
     • High fidelity audio output
     • Distribution of workload

                                       Mobile system

                        Audio
           Readfile     Decode          Playback
             Node        Node             Node

                                  Stationary system
                        Audio
                        Decode          Playback
                         Node             Node

                              MUM2003 - Dynamic Distributed Multimedia – p.14/21
                                                    Handover
•   Seamless and synchronized handover
     • No loss, no duplicates, no interruption

•   Idea
      • Create reconfigured copy (slave) of running flow
        graph (master)
      • Simultaneously stream data in master and slave
      • Playback data only in master, slave is muted
      • Wait, until slave has caught up with master
      • Simultaneously switch from master to slave




                               MUM2003 - Dynamic Distributed Multimedia – p.15/21
                                                   Handover
•   Create reconfigured copy (slave) of running flow graph
    (master)
     • Here: location is reconfigured

•   Session sharing service
Running session (master)
                    Audio            C
    Readfile       Decode     Playback             Synchronizer
      Node           Node       Node
     mobile         mobile     mobile
Query (slave)
                    Audio
    Readfile       Decode     Playback
      Node           Node       Node
      mobile     stationary   stationary
                              MUM2003 - Dynamic Distributed Multimedia – p.16/21
                                                       Handover
•   Create reconfigured copy (slave) of running flow graph
    (master)
     • Here: location is reconfigured

•   Session sharing service
Shared running session (master)
                       Audio             C
    Readfile group
              jack
                      Decode      Playback             Synchronizer
      Node             Node         Node
      mobile           mobile       mobile
Running session                          C
(slave)                Audio
                      Decode      Playback
                       Node         Node
                     stationary   stationary
                                  MUM2003 - Dynamic Distributed Multimedia – p.16/21
                                                                     Handover
•    Simultaneously stream data in master and slave
•    Playback data only in master, slave is muted
•    Wait, until slave has caught up with master
      • Controller of slave will notify synchronizer

Mobile system
                           2                1                           Audio
                       B        Audio   B              C
    Readfile   jack                             Playback               Handover
      Node
               group           Decode                                Synchronizer
                                Node              Node

Stationary system
                                Audio                  C
                               Decode           Playback
                                Node              Node


                                                MUM2003 - Dynamic Distributed Multimedia – p.17/21
                                                                   Handover
•   Simultaneously stream data in master and slave
•   Playback data only in master, slave is muted
•   Wait, until slave has caught up with master
     • Controller of slave will notify synchronizer

Mobile system
                               4          3                             Audio
                       B            Audio B S        C
    Readfile   jack                           Playback                Handover
      Node
               group               Decode                           Synchronizer
                           4        Node        Node
                       B
                               3
Stationary system          B
                                    Audio S           C
                                   Decode      Playback
                                    Node         Node


                                              MUM2003 - Dynamic Distributed Multimedia – p.17/21
                                                                 Handover
•   Simultaneously stream data in master and slave
•   Playback data only in master, slave is muted
•   Wait, until slave has caught up with master
     • Controller of slave will notify synchronizer

Mobile system
                           6                5                         Audio
                       B       Audio    B              C
    Readfile   jack                             Playback            Handover
      Node
               group           Decode                             Synchronizer
                                Node              Node
                           6
                       B
Stationary system
                                            5
                               Audio    B              C
                               Decode           Playback
                                Node              Node


                                            MUM2003 - Dynamic Distributed Multimedia – p.17/21
                                                                   Handover
•   Playback data on both system simultaneously, or
•   Simultaneously switch from master to slave
     • Instream events (mute ’M’ and un-mute ’U’)
     • Timed notification of sink nodes

Mobile system
                         6                5                           Audio
                     B       Audio    B              C
 Readfile    jack                             Playback              Handover
   Node
             group           Decode                               Synchronizer
                              Node              Node
                         6
                     B
Stationary system
                                          5
                             Audio    B              C
                             Decode           Playback
                              Node              Node



                                              MUM2003 - Dynamic Distributed Multimedia – p.18/21
                                                                   Handover
•   Playback data on both system simultaneously, or
•   Simultaneously switch from master to slave
     • Instream events (mute ’M’ and un-mute ’U’)
     • Timed notification of sink nodes

                                     switchOutputs

Mobile system
                                                     C                Audio
 Readfile                   Audio         M                          Handover
            jack
            group           Decode            Playback             Synchronizer
   Node                                         Node
                             Node
                        8
                    B
Stationary system
                                      7      C
                            Audio B U
                            Decode    Playback
                             Node       Node

                                              MUM2003 - Dynamic Distributed Multimedia – p.18/21
                                                                 Handover
•   Playback data on both system simultaneously, or
•   Simultaneously switch from master to slave
     • Instream events (mute ’M’ and un-mute ’U’)
     • Timed notification of sink nodes

Mobile system
                           6                5                         Audio
                       B       Audio    B              C
    Readfile   jack                             Playback            Handover
      Node
               group           Decode                             Synchronizer
                                Node              Node
                           6
                       B
Stationary system
                                            5
                               Audio    B              C
                               Decode           Playback
                                Node              Node



                                            MUM2003 - Dynamic Distributed Multimedia – p.18/21
                                                                 Handover
•   Playback data on both system simultaneously, or
•   Simultaneously switch from master to slave
     • Instream events (mute ’M’ and un-mute ’U’)
     • Timed notification of sink nodes

                                                                     7
                                                              mute

Mobile system
                           8                7                        Audio
                       B        Audio   B              C
    Readfile   jack                             Playback            Handover
      Node
               group           Decode                             Synchronizer
                                Node              Node
                           8
                       B
Stationary system
                                            7
                               Audio    B              C                     7
                               Decode           Playback            unmute

                                Node              Node

                                            MUM2003 - Dynamic Distributed Multimedia – p.18/21
                                               Conclusions
•   Network-Integrated Multimedia Middleware (NMM)
     • Distributed synchronization
     • Registry service

•   Session sharing as middleware service
     • Automatic sharing of (parts of) active flow graphs
     • Eliminates the problem of multiple access of
       hardware devices
     • Reduces the processing and networking
       requirements through re-usage
•   Session handover
     • Seamless and synchronized reconfiguration

                              MUM2003 - Dynamic Distributed Multimedia – p.19/21
                                              Future Work
•   Usage of multicast networking for shared sessions
•   Quality of Service measurement to improve value
    function
•   Seamless reconfiguration/migration of sessions with
    audio and video
•   Evaluation of different application scenarios
•   User interfaces to select shared sessions or
    reconfigurations




                                MUM2003 - Dynamic Distributed Multimedia – p.20/21
                  Questions?




MUM2003 - Dynamic Distributed Multimedia – p.21/21

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:0
posted:3/30/2012
language:English
pages:36