ppt

Document Sample
ppt Powered By Docstoc
					     Crossing Media for Video Search:
enabling usability beyond traditional broadcast & TV


        Katerina Pastra and Stelios Piperidis


        Language Technology Applications,
   Institute for Language and Speech Processing,
                   Athens, Greece
     The “Pervasive Digital Video” Era 

 TV sets extended with “intelligent” DVRs, set-top
boxes with PC-like functionalities, linked with PCs that
display streamed video and allow interaction through
gaming consoles
 Video viewing transferred beyond the TV set, to
mobiles & i-pods allowing on-the-move viewing
 Video broadcast carried through broadband using IP
 Video content (professional and/or consumer-generated) exchanged
through file swapping and headline syndication
technology
             A New Era in Video Search ?

  From the digital video libraries context to the new
            pervasive digital video reality:
The scope of video search (indexing & retrieval)
technologies is broadened and their role is reinforced


 Does pervasive digital video affect the “search” in
video search technologies? (imposes new challenges)
 Does video search affect the “pervasiveness” of
digital video? (affects usability of available video and
corresponding new technologies)
                       Overview

 Video search: the market perspective
   - market players and video search developers
   - video search in commercial prototypes
 Video search research prototypes
   - lessons from the digital video library scenarios
 New technological challenges
 Dealing with new challenges
   - Suggestions from the REVEAL THIS project
   - Using cross-media decision mechanisms
           Video Search & the Market Players
                         IPTV software developers




Content owners
                                                  ISP, computer
                                               networking & phone
                             Video search
                                                    companies
                         software developers


TV service providers &
file-swapping networks                                         Electronics Manufacturers

   Transitive
   Dependencies        Web content Aggregators,
   Dependency          content service providers,
   Trends            content repackaging companies
  Video search mechanisms in the market

Characteristics :
 Use of owner/broadcaster created metadata
 Text-based search on closed captions or         Not robust
  ASR or speech stream
                                                     Find the
 Processing of English files mostly languages?
                                            Other      right
                                                     keyword
 Keyword query (restricted semantic expansion)      problem!
 Retrieval unit is either whole video or short segment
  where keyword appears (+ few seconds before and after)
    Quest for
    coherence           Are such mechanisms efficient?
                  Video Search in Research
 The digital video library access scenarios prevailed
in research projects up until the ’00s (cf. Informedia, TRECVid etc.)
 Video indexing & retrieval prototypes explored a
variety of unimodal and multimodal mechanisms that
go beyond commercially offered video search
                                                    MM approaches slightly
Lessons learned      (Hauptmann and Christel 2004):  better. Necessary?

 Fusion of medium-specific retrieval results boosts
video retrieval performance slightly (vs. e.g. text-based only retrieval)
 Fusion based on linear weights ~ query-type helpful
 Text query/ASR enhancement, relevance feedback
and feature-concept associations, all helpful
  Video search challenges in the new context
 From advanced computer users to laymen
   - type & quality of query
   - expectations & requirements on retrieval accuracy
   - length of retrieval unit
   - domain and language of data
 From structured data collections to pervasive video
  - genre, domain, language, source/structure variation
  - consumer-generated / noisy / low quality and prof.
  - broadcast metadata, closed captions etc. availability
 From static to dynamic search
  - VoD & real-time broadcast data
  - re-active & pro-active, personalised search
    (push and pull)
                        The ideal search mechanism?
     Video search prototypes in the ’00s
 Multimedia approaches for video search suggested by
research projects with application scenarios related to
the new pervasive digital video context, e.g. leisure and
entertainment in the digital home, and/or for the
mobile user (e.g UP-TV, BUSMAN, AceMedia)
 Image features – language concepts association for
video search suggested (Multimedia integration)
 REVEAL THIS goes a step further in suggesting
the use of cross-media decision mechanisms
FP6 funded project (Nov. 2004-April 2007)
http://www.reveal-this.org
         REVEAL THIS Use Scenario

        WEB             TV            Radio
                                                               TV, Radio, Web data

                                                          A system that offers both types
                                          Media               of service :
                                         Archive
                                                          a)     Multimedia and Cross
         Media Server                                            lingual Information
          (Content      Reveal-THIS
         Aggregator)    technology                               Retrieval (pull)
                                                          b)     Multimedia and Cross
                                                                 lingual information
                               Search                            Filtering (push)
                  Delivery     archive
                                                                 Mobile phone,
                                                                 and Web interfaces

                                                                 EN-EL,
       User                                                      European Parliament plenary
                              Web                                sessions & press-conferences,
User   profile          Mobile                Local Archive
                                                                 national news, travel
                                                                 documentaries & info
                        Web text    radio         TV


                              Media Manager            video

     text
                            audio
                                      IAC – keyframe
                                     extraction, Image                    FDIC - face
  TPC - text              SPC–speech Analysis & image
                 text                                     keyframes       Detection &
  processing              processing   categorization                    identification


                     Automatically extracted metadata:
                      TPC: named entities, terms, facts
    Text                       Text Categories
categorisation     SPC: speaker turns, speaker names, text
                  IAC: shotcuts, keyframes, image features,
                              image categories
                          FDIC: face regions, names
                                                                          Translation

                         Story Boundary Detection
                                                                 Media Server:
                            cross-media stories                     Storage
                                                                Personalisation
                          Cross-media Indexing,                     Browse
                        Cross-media Categorization                   Query
                        Multimedia Summarization                   Retrieve
                                                               Push Notifications
      The notion of Cross-Media Decision
                 Mechanisms
Mechanisms that decide on the relation that holds
between medium specific pieces of information:
 across documents (Boll et al. 1999)
 within documents (Pastra 2006)

The mechanisms decided whether medium-specific
pieces of information within the same Multimedia
Document are:                          equivalence
 associated (multimedia integration)
 complementary      complementarity

 semantically compatible/incompatible
                    Conclusions

 The scope of video search technology is broadened & new
technological challenges are imposed Efficient video search
 The market players consider video search technology
indispensable                   is indispensable for usability
 Commercial video search limited; research in the digital
library access context goes beyond such limitations &
points to slight benefits in using multimedia fusion
techniques             beyond traditional broadcast and TV;
 Research with new application scenarios (iTV etc.)
emphasizes the necessity of such mechanisms & introduces
the notion of x-media decision mechanisms may hold the key
                                             for achieving it

				
DOCUMENT INFO