VIEWS: 0 PAGES: 59 POSTED ON: 10/5/2012
Lecture 26: Single-Image Super-Resolution CAP 5415 Announcements Projects Due Dec 2 If you have nice results and would like to make a 20 minute presentation on your project in class on Wednesday, let me know Bonus – reduction in page requirements for writeup The Goal of Super-Resolution We have a low-resolution version of an image We want to create a higher-resolution version Why not just use Photoshop? • Standard interpolation (bicubic or bilinear) create an image with more pixels, but don’t introduce new frequency content Small Sharp Picture Large Smooth Picture View in the Spatial Frequency Domain Original Image Doubled in Size Quadrupled in Size Each image shows the magnitude of the DFT of the image at a particular size. Interpolation cannot introduce high-frequency content into the high-res image Our goal is to create a system that can Today I will begin by talking about one of most frequently-cited super-resolution papers Then talk about my own extensions Our approach We take a probabilistic approach Create a distribution - high-resolution image - observed low-resolution image Find the high resolution image by finding the image that maximizes Other Approaches Level set approaches (Morse and Schwartzwald 2001) Non-linear enhancement (Greenspan 2000) Deblurring by inverting the convolution filter Unable to introduce new high-frequency components The Real Problem We can now find the high-resolution image by finding the choosing the right patch to put in at each point Pick patch from image database Basic Setup Low-Resolution Observation Bi-linearly interpolated Actual High-Res Store examples of corresponding patches (Images from Freeman et al.) The Real Problem We can now find the high-resolution image by finding the choosing the right patch to put in at each point Pick patch from image database Problem: That's a lot of patches! (Part) Solution 1: Filter out low-frequencies Reduces Variability (Part) Solution 2: Contrast Normalize (Takes out scale variability) What is really being stored Store these high-frequency pairs Getting a new image Look at low-resolution patch in image Find most similar low-res patch in database Fill in corresponding high-res Does this work? Correct High Res Answer What Happened? To Do Better This method treats each patch independently We need to consider patches together For example Edges tend to continue over long distances Modeling For a 256x256 image with 10 candidate patches, could have as many as 10256x256 numbers in the distribution Need to simplify our model! Strategy for simplification 1.Simplify relationships between pixels by using a Markov Random Field Strategy: Markov Random Field Assuming that a patch depends on its four nearest neighbors Naively, only need 10,000 numbers (big reduction!) The model Divide high-res image into 7x7 patches Our Model Divide high-res image into 7x7 patches Each patch is one node in the graph Or, each patch is one random variable in the distribution Another View Intuitively, we want to create a high-resolution image out of a collection of patches (Figure from Freeman) Choosing The compatibility between patches is determined by how similar their borders are Patch 1 Patch 2 Results First test, how well does it match the statistics of high-frequency, even if they're weird? Results Training Images: Results Failure Revisiting The Issue of States In this algorithm, the state of each node is which candidate patch to use Tradeoff: Too many states – Maximizing P(h|l) is intractable Too few states – unable to represent high-res image well with a small number of discrete states. Solution: Use the observed low-resolution image to choose a small number of patches that are likely to represent the high-res image well Choosing Candidates Using a database Have to store database Have to search database Different Approach: Generate the candidates directly from the low-res observation. 1-D Example Learn a set of interpolators Each interpolator creates a candidate high-resolution signal from the low-resolution input Where does MS come from? Use a training database of high-resolution/low- resolution pairs to choose the best interpolators Find the interpolators using a clustering algorithm: 1. Cluster the pairs using k-means clustering 2. For each cluster, find the interpolator that best predicts the high-resolution patches from the low- resolution patches 3. For each pair, reassign it to the cluster whose interpolator best predicts the high-resolution patch from the low-resolution patch 4. Repeat Steps 2 and 3 until convergence How Many Interpolators are Needed? Can use the training set to estimate a lower bound on the error incurred using the interpolators For super-resolution, we used 64 Choosing We use the image derivatives of neighboring patches to compute Red Pixels – Border of Patch 1 Blue Pixels – Border of Patch 2 Patch 1 Patch 2 The difference between a pair of adjacent red and blue pixels <1 Justifying We can justify this choice in two ways Image statistics Image “sharpness” Image Statistics and The distribution of the derivatives of a natural image is modeled well by where typically 0.7 < < 1.2 Using attempts to respect these statistics Image Sharpness Consider the problem of interpolating y1 from y0 and y2 Model the distribution of y1 as If α > 1, then the most likely value of y1 is 1.5 “Blurry” Edge If α < 1, then the most likely value of y1 is either 1 or 2 Sharp edge α < 1 acts as a sharpness prior. It prefers sharp edges Image Sharpness The compatibility function works the same way It favors setting the patches so that there are as few derivatives possible If there must be image derivatives, then If α > 1, then the compatibility function favors many small derivatives over a few large derivatives If α < 1, the compatibility function favors a few large derivatives over many small derivatives If α < 1 favors a high-res image with sharp edges Finding a high-resolution image Now that we have distribution, we can find a high-resolution image by maximizing One problem, With more than 2 states per node, maximizing is NP-Complete We're using 64 states per node! Maximizing Maximizing P(h|l) is intractable, so we have to use an approximate technique Approximate technique – Not guaranteed to find the best value of h, but we hope it finds a pretty good value Two Popular Techniques 1.Loopy Belief Propagation (Pearl 88, Weiss 98, Freeman 2000) 2.Graph Cuts (Boykov, Veksler, Zabih 2000) Loopy Belief Propagation Assume P(h|l) can be represented by a graph with no loops Can maximize P(h|l) using an algorithm based on passing messages between nodes in the graph (Pearl 88, Weiss 98) Messages encode belief about the state of each nodes Graph Cuts Start with an initial labeling of the graph Denote the probability of this labelling P(h0|l) How can we relabel the graph and increase P(h0|l)? 1 2 2 2 1 1 1 3 3 3 3 2 1 1 1 1 Graph Cuts Perform a swap: For two states s1 and s2, perform a swap by changing some nodes from s1 to s2 and vice-versa 1 2 2 2 1 1 2 2 1 1 1 3 1 1 1 3 3 3 3 2 3 3 3 2 1 1 1 1 1 2 2 1 The optimal swap can be found in polynomial time Graph Cuts Keep performing swaps between states until you can no longer increase P(h|l) with a single swap P(h|l) never decreases! Practically, convergence is guaranteed Which should I use? Belief Propagation Graph Cuts No guarantees about No guarantees about maximizing P(h|l) maximizing P(h|l) Not guaranteed to Guaranteed to converge converge Can handle arbitrary Not sure graphs Cannot produce Can estimate the marginals marginal probabilities Tends to find better of P(h|l) solutions than BP caveat: these (Tappen 2003) marginals will be wrong Which did we use? Our model also includes a reconstruction constraint that forces the recovered high- resolution image to match the low-resolution image when it is down-sampled The peculiar graph structure led us to use belief propagation (Already had BP code too) BP code available at http://www.ai.mit.edu/~mtappen Results Actual High Res Pixel Replicated Bicubic Interpolation Low-Res Greenspan et al. Our method Results Actual High Res Pixel Replicated Bicubic Interpolation Low-Res Greenspan et al. Our method CCD Demosaicing This approach is flexible enough to be applied to other image processing problems We also applied it to CCD Demosaicing Common Artifacts Results Results