multimedia

Document Sample
multimedia Powered By Docstoc
					Multimedia and Time-series Data
• Today we need to query and analyze vast amounts
of multimedia data (i.e., images, sound tracks, video
tracks).
• Joint Research from Database
Management, Computer Vision and Signal Processing
aims to solve problems related to multimedia data
management.
• Today we will see an overview of some techniques
used for this problems.
• We will also briefly discuss how time-series data are
managed in database systems.


Dr. N. Mamoulis   Advanced Database Technologies          1
Multimedia Data
• There are four major types of multimedia data:
images, video tracks, sound tracks, and text.
• From the above, the easiest type to manage is text,
since we can order, index, and search text using
string management techniques.
• Management of simple sounds is also relatively
easy, since they can be represented by signal
sequences over channels.
• We will describe techniques for image retrieval.
These can are extended and applied also for video
retrieval.


Dr. N. Mamoulis   Advanced Database Technologies        2
Content-based Image Retrieval
• Images were traditionally managed by first
annotating their contents and then using text-
retrieval techniques to index them.
• However, with the increase of information in digital
image format some drawbacks of this technique were
revealed:
      • Manual annotation requires vast amount of labor
      • Different people may perceive differently the contents of
      an image; thus no objective keywords for search are defined
• A new research field was born in the 90’s: Content-
based Image Retrieval aims at indexing and retrieving
images based on their visual contents.

Dr. N. Mamoulis       Advanced Database Technologies                3
Feature Extraction
• The basis of Content-based Image Retrieval is to
extract and index some visual features of the images.
• There are general features
(e.g., color, texture, shape, etc.) and domain-specific
features (e.g., objects contained in the image).
      • Domain-specific feature extraction can vary with the
      application domain and is based on pattern recognition
      • On the other hand, general features can be used
      independently from the image domain.
• Next, we will see how the general features of
images are indexed and queried in a database.



Dr. N. Mamoulis        Advanced Database Technologies          4
Color Features
• The color is the dominant general visual feature of an image.
Humans primarily distinguish images based on the color
proportions in them.
• There are three types of receptors in our eyes. They receive
red, green and blue colors. This is the motivation of the RGB
model. Adding these three primary colors together produces the
rest of known colors. An image can be represented by a two-
dimensional array of color values (pixels).




      pixel representation                            RGB color space
Dr. N. Mamoulis              Advanced Database Technologies             5
Color Features (cont’d)
• To represent the color of an image compactly, a color
histogram is used. Colors are partitioned to k groups according
to their similarity and the percentage of each group in the image
is measured.
• Images are transformed to k-dimensional points and a
distance metric (e.g., Euclidean distance) is used to measure the
similarity between them.
                                                      k-dimensional space




                                  k-bins

Dr. N. Mamoulis      Advanced Database Technologies                  6
Color Features (cont’d)
• A color space which facilitates quantization of color distribution
in an image is HSV
    • Hue: what the color is. This tells where along the
    spectrum your color lies.
    • Saturation: how pure the color is (high values -> stronger
    color small values -> faded color).
    • Value: how bright the color is.

           RGB                         HSV




Dr. N. Mamoulis       Advanced Database Technologies                   7
Texture and Shape
• Texture refers to the visual patterns that are homogeneous
and do not result from the presence of one color or intensity
(e.g., clouds, bricks, hair, fabric, etc.).
• Several representations of texture have been considered,
among them a co-occurrence matrix for spatial color
dependencies, visual texture properties (coarseness, regularity,
etc.), and wavelet transforms of texture images.
• The boundaries of shapes are also indexed using transforms
(e.g., Fourier Descriptors, Moment Invariants). We will see later
various transforms used for approximation and indexing.




Dr. N. Mamoulis       Advanced Database Technologies                8
Color Layout
• Querying by global color may give too many false results in
large image collections.

                                                      100% similar
                                                      by global color




• Images are usually divided into blocks and color features are
extracted from each of the blocks. Similarity is then aggregated
from all blocks.
• More sophisticated techniques for image partitioning have
been proposed (e.g., space partitioning using quadtrees)

Dr. N. Mamoulis      Advanced Database Technologies                     9
Time-series Data
• A time-series is a sequential collection of events
over time.
• Time series data are found in various fields, e.g.,
stock market values, sensor indications, cardiograms.
• There are several search problems over time-series
data, e.g.:
      • find (approximate/exact) occurrences of a query
      subsequence q in a long sequence T.
      • find the most similar sequence to a query sequence q
      • classify/cluster sequences based on their similarity
      • find frequent event patterns in long sequences
      • ...


Dr. N. Mamoulis        Advanced Database Technologies          10
Common Search Problems in Multimedia Data
and Time-series Data
• Comparing long sequences, or long feature vectors
is time consuming.
• Indexing them as points in a high dimensional
space does not work due to the ‘curse of
dimensionality’; the efficiency of spatial access
methods degrades fast with dimensionality and the
entropy of high dimensional space makes NN search
less meaningful (more in presentation 1...)
• The problem can be alleviated by:
      • data approximation and dimensionality reduction
      • data compression and linear scan



Dr. N. Mamoulis       Advanced Database Technologies      11
Using Transformations to Reduce
Dimensionality
• In many cases the embedded dimensionality of a search
problem is much lower than the actual dimensionality
• Some methods apply transformations on the data and
approximate them with low-dimensional vectors
• The aim is to reduce dimensionality and at the same time
maintain the data characteristics
• If d(a,b) is the distance between two objects a, b in real (high-
dimensional) and d’(a’,b’) is their distance in the transformed
low-dimensional space, we want d’(a’,b’)d(a,b).


                                  d’(a’,b’)

          d(a,b)




Dr. N. Mamoulis       Advanced Database Technologies                  12
Example: Discrete Fourier                                                                        X



Transform (DFT)                                                                                  X'




• How to represent a time-series (or a color
                                                            0   20   40   60   80   100   120    140
histogram) using only n numbers (in a n-
dimensional space)?                                                                              0



• Basic Idea: Represent the time series as a                                                     1


linear combination of sines and cosines, but                                                     2

keep only the first n/2 coefficients.
                                                                                                 3



Why n/2 coefficients? Because each wave requires                                                 4


2 numbers, for the phase (w) and amplitude (A,B).                                                5



                                                                                                 6




           n                                                                                     7


  C (t )   ( Ak cos(2wk t )  Bk sin(2wk t ))                                                8
          k 1
                                                                                                 9




Dr. N. Mamoulis            Advanced Database Technologies                                   13
Using Transformations to Reduce
Dimensionality (cont’d)
• Other popular transformations include:
    • Discrete Wavelet Transform. The sequence is transformed
    to a linear combination of Wavelet basis functions, but only
    the first n coefficients are used.
    • Singular Value Decomposition. Similarly, the sequence is
    transformed to a linear combination of eigenwaves, and only
    the first n coefficients are used.
    • Piecewise approximations are also used for time-series
    data.

• In presentation 2 we will see one of the most commonly used
methods for dimensionality reduction (FastMap). In presentation
3 we will see how the problem is solved for another similarity
metric, the dynamic time warping.

Dr. N. Mamoulis      Advanced Database Technologies                14
Using Transformations to Reduce
Dimensionality (cont’d)
• After the transformation, the Euclidean distance of the vectors
in the low-dimensional space lower bounds the actual distance
in the high-dimensional space. If d(a,b) is the distance between
two objects a, b in real (high-dimensional) and d’(a’,b’) is their
distance in the transformed low-dimensional space, then
d’(a’,b’)d(a,b).

• A spatial access method is used (e.g., R-trees) to index the
low-dimensional vectors and a NN-search algorithm (e.g., INN)
to perform similarity search.

                           d’(a’,b’)

          d(a,b)




Dr. N. Mamoulis       Advanced Database Technologies                 15
Using Transformations to Reduce
Dimensionality (cont’d)
Assume that we want to find most similar image (or time-series) to
a query q.
1. q is transformed to q’ in the low dimensional
   space.                                                      q
                                                       o
2. INN is performed using the R-tree and the               d(q,o)
   nearest neighbor o’ is retrieved.
3. The distance between the (real) high-
   dimensional vectors is computed d(q,o).
4. We know that no object p with d’(q’,p’)>d(q,o)
   can be the NN because d(q,p)d’(q’,p’)>d(q,o)      d’(q’,o’)
                                                          o’ q’ d(q,o)
5. A range query is applied on the R-tree using
   d(q,o) in the low dimensional space, to get all
   objects p with d’(q’,p’)<d(q,o).
                                                       candidates
6. The real distance d(q,p) for each p is
   computed and the closest p is the actual NN.
Dr. N. Mamoulis      Advanced Database Technologies                      16
Searching for Similar Subsequences in a Long
Sequence

                  q:      T:
                                                            |T|- s windows
• We index the long sequence T as follows.              s
• We slide a window of size s at each position of T
• We apply a transformation (e.g., DFT) to
approximate the subsequence at each of the |T|-s
positions.
• We index the transformations using a SAM
(e.g., R-tree)
• Use the tree to search for subsequences
which are close to the query subsequence.
The results are approximate, so they have to
be validated by accessing the exact data


Dr. N. Mamoulis        Advanced Database Technologies                17
Summary
• Multimedia data can be represented by a set of feature
values in the high-dimensional space.
• Time-series data are event sequences in time.
• Similarity search is expensive in high dimensionality;
the data are usually approximated by low dimensional
points, so that their distances ‘lower bound’ their actual
high dimensional distances.
• The low dimensional vectors are indexed by spatial
access methods and NN search facilitates fast
approximate similarity search.
• We will now see why in high dimensional spaces
search is often meaningless, a dimensionality reduction
method and a time-series similarity technique.
Dr. N. Mamoulis   Advanced Database Technologies       18

				
DOCUMENT INFO