Research Projects
6v81 Multimedia Database
Yohan Jin, T.A.
Content Based Image Annotation
• Problem
▫ Huge Images still need to be annotated
▫ Flicker – 4,116 photos (10:07~10:08 pm 01/15/2008)
• Goal
▫ Annotate Automatically with Test-Images
• DataSet
▫ Corel DataSet (4,500:Training,500:Test)
• Approaches
▫ Low-level Feature Tracking (color,texture,shape)
▫ Data-Mining Algorithms (SVM,CMRM)
• Related Publication
▫ “Automatic image annotation and retrieval using cross-
media relevance models”-ACM SIGIR2003
Image Annotation Refinement
• Problem
▫ Semantic Gap- classification accuracy < 30%
• Goal
▫ Enhance classification accuracy using KB
▫ Apply Semantic Distance
• DataSet
▫ Coral DataSet
• Approaches
▫ WordNet Semantic Distance program
• Related Publication
▫ “Image annotations by combining multiple evidence &
wordNet”, ACM MM05’
Web-Image Annotation Refinement
• Problem
▫ Try to use surrounding text as the annotation of web-image
▫ But, there are quite un-related keywords
• Goal
▫ Remove un-related keywords
▫ Annotate web-images with un-removed ones
• DataSet
▫ Any web-page directory (e.g., CNN.com/travel)
• Approaches
▫ Web-image Crawler, Google/WordNet Semantic Distance
• Related Publication
▫ “Bipartite Graph Reinforcement Model for Web Image
Annotation”, ACM MM07’
Video Human Gesture Recognition
• Problem
▫ Video-input is quite cheap and useful
• Goal
▫ Recognize automatically test-vide0 human motion
• DataSet
▫ Several kinds of small motion clips(Test,Train)
• Approaches
▫ openCV human motion tracking
▫ Data-Mining Algorithm (SVM, GaussianMixture)
• Related Publication
Video Gesture Recognition
Enhancement using 3D Mocap Data
• Problem
▫ Sometimes, video human motion is quite noisy
▫ Use similar 3D motion Capture Data
• Goal
▫ Enhance the recognition rate using 3D similar
motion data
• DataSet
▫ Video motions, captured similar 3D mocap data
• Approach
▫ openCV video tracking, HMM
3D Human Motion Classification
• Problem
▫ Try to classify 3D mocap Data
▫ It is multidimensional Time series Data
• Goal
▫ Automatically classify or cluster 3D mocap Data
• DataSet
▫ CMU mocap (2,265), UTD mocap
• Approach
▫ Matlab, Data-Mining Algorithm
• Related Publications
▫ Segmentation and Recognition of Motion Streams by Similarity
Search- ACM TOMCCAP
▫ Semantic Quantization of 3D Human Motion Data Through
Spatial-Temporal Feature Extraction – MMM 2008
P2P Streaming
• Problem
▫ P2P continuous media streaming
• Goal
▫ Streaming the continuous media like video or audio over wired or
wireless networks.
• Approach
▫ In order to guarantee a high quality at end users, we have to keep
the transmission delay as small as possible. At the same time, we
may face the distortion caused by packet loss during
transmission. In p2p streaming, each peer contributes its upload
bandwidth to redistributing the data stream. How to utilize the
upload bandwidth of each peer and how to balance different
bandwidth are important in this project. If streaming by wireless
networks, we need to consider how to deal with very limited
bandwidth and high packet loss. An example of this project could
be sharing a video clip between some users with different
bandwidths.
Collaborative Platform
• Problem
▫ Streaming with multi-users collaborative interaction
• Goal
▫ Building a collaborative work environment for distributed
multimedia applications.
• Approach
▫ We focus on how multiple users can manipulate the same object
(image, 3D model, or video) in real time. We should support
working in both synchronous and asynchronous ways. Moreover,
effective conflicts detection and data streaming to realize the real-
time are important in this project. An example of this project
could be an environment for multi-users to draw a picture at the
same time.
3D Streaming
• Problem
▫ Streaming 3D Models
• Goal
▫ Stream 3D models in a progressive way.
• Approach
▫ Use multi-layer, progressive representation of 3D models.
Identify data set that can be transmitted over unreliable channel
and those that need reliable channel. Stream the data
appropriately.
Haptic Streaming
• Problem
▫ Stream data from haptic devices
• Goal
▫ Stream haptic data, receive force feed back, and show.
• Approach
▫ Haptic data is typically generated at 1000 samples per second.
This is to be streamed to a “server”. Server will send force
feedback. This is to be applied back to the device.
▫ Typically used in “tele” applications – telemedicine, telerobotics,
etc.
Indexing Structures
• Problem
▫ Search through multimedia databases
• Goal
▫ Focus on some new media type – say, 3D models. Develop index
structures for fast search.
• Approach
▫ “Generate “ features from 3D models. Index these features using
spatial index structures.
Mining Multimedia Data
• Problem
▫ Identify patterns of “events” in multimedia data
• Goal
▫ Focus on new media type – 3D, medical data
• Approach
▫ Use association rule mining to identify patterns.