VIEWS: 88 PAGES: 18 POSTED ON: 4/30/2010
Multimedia and Time-series Data • Today we need to query and analyze vast amounts of multimedia data (i.e., images, sound tracks, video tracks). • Joint Research from Database Management, Computer Vision and Signal Processing aims to solve problems related to multimedia data management. • Today we will see an overview of some techniques used for this problems. • We will also briefly discuss how time-series data are managed in database systems. Dr. N. Mamoulis Advanced Database Technologies 1 Multimedia Data • There are four major types of multimedia data: images, video tracks, sound tracks, and text. • From the above, the easiest type to manage is text, since we can order, index, and search text using string management techniques. • Management of simple sounds is also relatively easy, since they can be represented by signal sequences over channels. • We will describe techniques for image retrieval. These can are extended and applied also for video retrieval. Dr. N. Mamoulis Advanced Database Technologies 2 Content-based Image Retrieval • Images were traditionally managed by first annotating their contents and then using text- retrieval techniques to index them. • However, with the increase of information in digital image format some drawbacks of this technique were revealed: • Manual annotation requires vast amount of labor • Different people may perceive differently the contents of an image; thus no objective keywords for search are defined • A new research field was born in the 90’s: Content- based Image Retrieval aims at indexing and retrieving images based on their visual contents. Dr. N. Mamoulis Advanced Database Technologies 3 Feature Extraction • The basis of Content-based Image Retrieval is to extract and index some visual features of the images. • There are general features (e.g., color, texture, shape, etc.) and domain-specific features (e.g., objects contained in the image). • Domain-specific feature extraction can vary with the application domain and is based on pattern recognition • On the other hand, general features can be used independently from the image domain. • Next, we will see how the general features of images are indexed and queried in a database. Dr. N. Mamoulis Advanced Database Technologies 4 Color Features • The color is the dominant general visual feature of an image. Humans primarily distinguish images based on the color proportions in them. • There are three types of receptors in our eyes. They receive red, green and blue colors. This is the motivation of the RGB model. Adding these three primary colors together produces the rest of known colors. An image can be represented by a two- dimensional array of color values (pixels). pixel representation RGB color space Dr. N. Mamoulis Advanced Database Technologies 5 Color Features (cont’d) • To represent the color of an image compactly, a color histogram is used. Colors are partitioned to k groups according to their similarity and the percentage of each group in the image is measured. • Images are transformed to k-dimensional points and a distance metric (e.g., Euclidean distance) is used to measure the similarity between them. k-dimensional space k-bins Dr. N. Mamoulis Advanced Database Technologies 6 Color Features (cont’d) • A color space which facilitates quantization of color distribution in an image is HSV • Hue: what the color is. This tells where along the spectrum your color lies. • Saturation: how pure the color is (high values -> stronger color small values -> faded color). • Value: how bright the color is. RGB HSV Dr. N. Mamoulis Advanced Database Technologies 7 Texture and Shape • Texture refers to the visual patterns that are homogeneous and do not result from the presence of one color or intensity (e.g., clouds, bricks, hair, fabric, etc.). • Several representations of texture have been considered, among them a co-occurrence matrix for spatial color dependencies, visual texture properties (coarseness, regularity, etc.), and wavelet transforms of texture images. • The boundaries of shapes are also indexed using transforms (e.g., Fourier Descriptors, Moment Invariants). We will see later various transforms used for approximation and indexing. Dr. N. Mamoulis Advanced Database Technologies 8 Color Layout • Querying by global color may give too many false results in large image collections. 100% similar by global color • Images are usually divided into blocks and color features are extracted from each of the blocks. Similarity is then aggregated from all blocks. • More sophisticated techniques for image partitioning have been proposed (e.g., space partitioning using quadtrees) Dr. N. Mamoulis Advanced Database Technologies 9 Time-series Data • A time-series is a sequential collection of events over time. • Time series data are found in various fields, e.g., stock market values, sensor indications, cardiograms. • There are several search problems over time-series data, e.g.: • find (approximate/exact) occurrences of a query subsequence q in a long sequence T. • find the most similar sequence to a query sequence q • classify/cluster sequences based on their similarity • find frequent event patterns in long sequences • ... Dr. N. Mamoulis Advanced Database Technologies 10 Common Search Problems in Multimedia Data and Time-series Data • Comparing long sequences, or long feature vectors is time consuming. • Indexing them as points in a high dimensional space does not work due to the ‘curse of dimensionality’; the efficiency of spatial access methods degrades fast with dimensionality and the entropy of high dimensional space makes NN search less meaningful (more in presentation 1...) • The problem can be alleviated by: • data approximation and dimensionality reduction • data compression and linear scan Dr. N. Mamoulis Advanced Database Technologies 11 Using Transformations to Reduce Dimensionality • In many cases the embedded dimensionality of a search problem is much lower than the actual dimensionality • Some methods apply transformations on the data and approximate them with low-dimensional vectors • The aim is to reduce dimensionality and at the same time maintain the data characteristics • If d(a,b) is the distance between two objects a, b in real (high- dimensional) and d’(a’,b’) is their distance in the transformed low-dimensional space, we want d’(a’,b’)d(a,b). d’(a’,b’) d(a,b) Dr. N. Mamoulis Advanced Database Technologies 12 Example: Discrete Fourier X Transform (DFT) X' • How to represent a time-series (or a color 0 20 40 60 80 100 120 140 histogram) using only n numbers (in a n- dimensional space)? 0 • Basic Idea: Represent the time series as a 1 linear combination of sines and cosines, but 2 keep only the first n/2 coefficients. 3 Why n/2 coefficients? Because each wave requires 4 2 numbers, for the phase (w) and amplitude (A,B). 5 6 n 7 C (t ) ( Ak cos(2wk t ) Bk sin(2wk t )) 8 k 1 9 Dr. N. Mamoulis Advanced Database Technologies 13 Using Transformations to Reduce Dimensionality (cont’d) • Other popular transformations include: • Discrete Wavelet Transform. The sequence is transformed to a linear combination of Wavelet basis functions, but only the first n coefficients are used. • Singular Value Decomposition. Similarly, the sequence is transformed to a linear combination of eigenwaves, and only the first n coefficients are used. • Piecewise approximations are also used for time-series data. • In presentation 2 we will see one of the most commonly used methods for dimensionality reduction (FastMap). In presentation 3 we will see how the problem is solved for another similarity metric, the dynamic time warping. Dr. N. Mamoulis Advanced Database Technologies 14 Using Transformations to Reduce Dimensionality (cont’d) • After the transformation, the Euclidean distance of the vectors in the low-dimensional space lower bounds the actual distance in the high-dimensional space. If d(a,b) is the distance between two objects a, b in real (high-dimensional) and d’(a’,b’) is their distance in the transformed low-dimensional space, then d’(a’,b’)d(a,b). • A spatial access method is used (e.g., R-trees) to index the low-dimensional vectors and a NN-search algorithm (e.g., INN) to perform similarity search. d’(a’,b’) d(a,b) Dr. N. Mamoulis Advanced Database Technologies 15 Using Transformations to Reduce Dimensionality (cont’d) Assume that we want to find most similar image (or time-series) to a query q. 1. q is transformed to q’ in the low dimensional space. q o 2. INN is performed using the R-tree and the d(q,o) nearest neighbor o’ is retrieved. 3. The distance between the (real) high- dimensional vectors is computed d(q,o). 4. We know that no object p with d’(q’,p’)>d(q,o) can be the NN because d(q,p)d’(q’,p’)>d(q,o) d’(q’,o’) o’ q’ d(q,o) 5. A range query is applied on the R-tree using d(q,o) in the low dimensional space, to get all objects p with d’(q’,p’)<d(q,o). candidates 6. The real distance d(q,p) for each p is computed and the closest p is the actual NN. Dr. N. Mamoulis Advanced Database Technologies 16 Searching for Similar Subsequences in a Long Sequence q: T: |T|- s windows • We index the long sequence T as follows. s • We slide a window of size s at each position of T • We apply a transformation (e.g., DFT) to approximate the subsequence at each of the |T|-s positions. • We index the transformations using a SAM (e.g., R-tree) • Use the tree to search for subsequences which are close to the query subsequence. The results are approximate, so they have to be validated by accessing the exact data Dr. N. Mamoulis Advanced Database Technologies 17 Summary • Multimedia data can be represented by a set of feature values in the high-dimensional space. • Time-series data are event sequences in time. • Similarity search is expensive in high dimensionality; the data are usually approximated by low dimensional points, so that their distances ‘lower bound’ their actual high dimensional distances. • The low dimensional vectors are indexed by spatial access methods and NN search facilitates fast approximate similarity search. • We will now see why in high dimensional spaces search is often meaningless, a dimensionality reduction method and a time-series similarity technique. Dr. N. Mamoulis Advanced Database Technologies 18