# My Talk by 3e6gVny

VIEWS: 0 PAGES: 59

• pg 1
```									Lecture 26: Single-Image Super-Resolution

CAP 5415
Announcements
   Projects Due Dec 2
   If you have nice results and would like to make a
20 minute presentation on your project in class on
Wednesday, let me know
   Bonus – reduction in page requirements for writeup
The Goal of Super-Resolution

We have a low-resolution
version of an image
We want to create a higher-resolution
version
Why not just use Photoshop?
• Standard interpolation (bicubic or bilinear) create
an image with more pixels, but don’t introduce
new frequency content

Small Sharp Picture

Large Smooth Picture
View in the Spatial Frequency Domain

Original Image             Doubled in Size          Quadrupled in Size

Each image shows the magnitude of the DFT of the image at a particular size.

   Interpolation cannot introduce high-frequency content into the high-res image

   Our goal is to create a system that can
Today
   I will begin by talking about one of most
frequently-cited super-resolution papers
   Then talk about my own extensions
Our approach
   We take a probabilistic approach
   Create a distribution
   - high-resolution image
   - observed low-resolution image
   Find the high resolution image by finding the
image that maximizes
Other Approaches
   Level set approaches (Morse and Schwartzwald
2001)
   Non-linear enhancement (Greenspan 2000)
   Deblurring by inverting the convolution filter
   Unable to introduce new high-frequency components
The Real Problem
   We can now find the high-resolution image by
finding the choosing the right patch to put in at
each point
   Pick patch from image database
Basic Setup

Low-Resolution Observation

Bi-linearly interpolated           Actual High-Res

Store examples of corresponding patches
(Images from Freeman et al.)
The Real Problem
   We can now find the high-resolution image by
finding the choosing the right patch to put in at
each point
   Pick patch from image database
   Problem:
   That's a lot of patches!
   (Part) Solution 1:
   Filter out low-frequencies
   Reduces Variability
   (Part) Solution 2:
   Contrast Normalize (Takes out scale variability)
What is really being stored

Store these high-frequency pairs
Getting a new image
   Look at low-resolution patch in image
   Find most similar low-res patch in database
   Fill in corresponding high-res
   Does this work?

Correct High Res                  Answer
What Happened?
To Do Better
   This method treats each patch independently
   We need to consider patches together
   For example
   Edges tend to continue over long distances
Modeling
   For a 256x256 image with 10 candidate patches,
could have as many as 10256x256 numbers in the
distribution
   Need to simplify our model!
   Strategy for simplification
1.Simplify relationships between pixels by using
a Markov Random Field
Strategy: Markov Random Field

   Assuming that a patch depends on its four nearest
neighbors
   Naively, only need 10,000 numbers (big
reduction!)
The model

   Divide high-res image into 7x7 patches
Our Model

   Divide high-res image into 7x7 patches
   Each patch is one node in the graph
   Or, each patch is one random variable in the
distribution
Another View

   Intuitively, we want to create a high-resolution
image out of a collection of patches

(Figure from Freeman)
Choosing
   The compatibility between patches           is
determined by how similar their borders are

Patch 1         Patch 2
Results
   First test, how well does it match the statistics of
high-frequency, even if they're weird?
Results
   Training Images:
Results
Failure
Revisiting The Issue of States
   In this algorithm, the state of each node is which
candidate patch to use
   Too many states – Maximizing P(h|l) is intractable
   Too few states – unable to represent high-res image
well with a small number of discrete states.
   Solution:
   Use the observed low-resolution image to choose a
small number of patches that are likely to represent
the high-res image well
Choosing Candidates
   Using a database
   Have to store database
   Have to search database
   Different Approach:
   Generate the candidates directly from the low-res
observation.
1-D Example

 Learn a set of interpolators
 Each interpolator creates a candidate high-resolution signal from the

low-resolution input
Where does MS come from?
   Use a training database of high-resolution/low-
resolution pairs to choose the best interpolators
   Find the interpolators using a clustering
algorithm:
1. Cluster the pairs using k-means clustering
2. For each cluster, find the interpolator that best
predicts the high-resolution patches from the low-
resolution patches
3. For each pair, reassign it to the cluster whose
interpolator best predicts the high-resolution patch
from the low-resolution patch
4. Repeat Steps 2 and 3 until convergence
How Many Interpolators are Needed?

   Can use the training set to estimate a lower bound
on the error incurred using the interpolators
   For super-resolution, we used 64
Choosing
   We use the image derivatives of neighboring
patches to compute
Red Pixels – Border of Patch 1
Blue Pixels – Border of Patch 2
Patch 1        Patch 2

The difference
between
a pair of adjacent
red and blue pixels

<1
Justifying

   We can justify this choice in two ways
   Image statistics
   Image “sharpness”
Image Statistics and

The distribution of the derivatives of a natural image is modeled
well by

where typically 0.7 <   < 1.2

Using                                 attempts to respect these
statistics
Image Sharpness

   Consider the problem of interpolating y1 from y0 and y2
   Model the distribution of y1 as
   If α > 1, then the most likely value of y1 is 1.5
   “Blurry” Edge
   If α < 1, then the most likely value of y1 is either 1 or 2
   Sharp edge
   α < 1 acts as a sharpness prior. It prefers sharp edges
Image Sharpness
   The compatibility function works the same way
   It favors setting the patches so that there are as
few derivatives possible
   If there must be image derivatives, then
   If α > 1, then the compatibility function favors many
small derivatives over a few large derivatives
   If α < 1, the compatibility function favors a few large
derivatives over many small derivatives
   If α < 1 favors a high-res image with sharp edges
Finding a high-resolution image
   Now that we have distribution, we can find a
high-resolution image by maximizing

   One problem,
   With more than 2 states per node, maximizing
is NP-Complete
   We're using 64 states per node!
Maximizing
   Maximizing P(h|l) is intractable, so we have to
use an approximate technique
   Approximate technique – Not guaranteed to find the
best value of h, but we hope it finds a pretty good
value
   Two Popular Techniques
1.Loopy Belief Propagation (Pearl 88, Weiss 98,
Freeman 2000)
2.Graph Cuts (Boykov, Veksler, Zabih 2000)
Loopy Belief Propagation
   Assume P(h|l) can be represented by a graph
with no loops

   Can maximize P(h|l) using an algorithm based
on passing messages between nodes in the graph
(Pearl 88, Weiss 98)

   Messages encode belief about the state of each
nodes
Graph Cuts
   Start with an initial labeling of the graph
   Denote the probability of this labelling P(h0|l)
   How can we relabel the graph and increase
P(h0|l)?
1    2     2    2

1    1     1    3

3    3     3    2

1    1     1    1
Graph Cuts
   Perform a swap:
   For two states s1 and s2, perform a swap by changing
some nodes from s1 to s2 and vice-versa

1    2     2    2             1    1    2    2

1    1     1    3             1    1    1    3

3    3     3    2             3    3    3    2

1    1    1     1            1     2    2    1

   The optimal swap can be found in polynomial time
Graph Cuts
   Keep performing swaps between states until you
can no longer increase P(h|l) with a single swap
   P(h|l) never decreases!
   Practically, convergence is guaranteed
Which should I use?
   Belief Propagation              Graph Cuts
   No guarantees about             No guarantees about
maximizing P(h|l)                maximizing P(h|l)
   Not guaranteed to               Guaranteed to
converge                         converge
   Can handle arbitrary            Not sure
graphs                          Cannot produce
   Can estimate the                 marginals
marginal probabilities          Tends to find better
of P(h|l)                        solutions than BP
   caveat: these               (Tappen 2003)
marginals will be
wrong
Which did we use?
   Our model also includes a reconstruction
constraint that forces the recovered high-
resolution image to match the low-resolution
image when it is down-sampled
   The peculiar graph structure led us to use belief
propagation
   BP code available at
http://www.ai.mit.edu/~mtappen
Results

Actual High Res           Pixel Replicated      Bicubic Interpolation
Low-Res

Greenspan et al.                 Our method
Results

Actual High Res           Pixel Replicated      Bicubic Interpolation
Low-Res

Greenspan et al.                 Our method
CCD Demosaicing
   This approach is flexible enough to be applied to
other image processing problems
   We also applied it to CCD Demosaicing
Common Artifacts
Results
Results

```
To top