My Talk by 3e6gVny


									Lecture 26: Single-Image Super-Resolution

               CAP 5415
   Projects Due Dec 2
   If you have nice results and would like to make a
    20 minute presentation on your project in class on
    Wednesday, let me know
       Bonus – reduction in page requirements for writeup
       The Goal of Super-Resolution

We have a low-resolution
version of an image
                           We want to create a higher-resolution
      Why not just use Photoshop?
• Standard interpolation (bicubic or bilinear) create
  an image with more pixels, but don’t introduce
  new frequency content

  Small Sharp Picture

                              Large Smooth Picture
         View in the Spatial Frequency Domain

       Original Image             Doubled in Size          Quadrupled in Size

Each image shows the magnitude of the DFT of the image at a particular size.

   Interpolation cannot introduce high-frequency content into the high-res image

   Our goal is to create a system that can
   I will begin by talking about one of most
    frequently-cited super-resolution papers
   Then talk about my own extensions
                  Our approach
   We take a probabilistic approach
   Create a distribution
       - high-resolution image
       - observed low-resolution image
   Find the high resolution image by finding the
    image that maximizes
                Other Approaches
   Level set approaches (Morse and Schwartzwald
   Non-linear enhancement (Greenspan 2000)
   Deblurring by inverting the convolution filter
       Unable to introduce new high-frequency components
               The Real Problem
   We can now find the high-resolution image by
    finding the choosing the right patch to put in at
    each point
   Pick patch from image database
                    Basic Setup

              Low-Resolution Observation

Bi-linearly interpolated           Actual High-Res

          Store examples of corresponding patches
                                           (Images from Freeman et al.)
                 The Real Problem
   We can now find the high-resolution image by
    finding the choosing the right patch to put in at
    each point
   Pick patch from image database
   Problem:
       That's a lot of patches!
   (Part) Solution 1:
       Filter out low-frequencies
       Reduces Variability
   (Part) Solution 2:
       Contrast Normalize (Takes out scale variability)
What is really being stored

   Store these high-frequency pairs
              Getting a new image
   Look at low-resolution patch in image
   Find most similar low-res patch in database
   Fill in corresponding high-res
   Does this work?

     Correct High Res                  Answer
What Happened?
                     To Do Better
   This method treats each patch independently
   We need to consider patches together
   For example
       Edges tend to continue over long distances
   For a 256x256 image with 10 candidate patches,
    could have as many as 10256x256 numbers in the
   Need to simplify our model!
   Strategy for simplification
    1.Simplify relationships between pixels by using
     a Markov Random Field
     Strategy: Markov Random Field

   Assuming that a patch depends on its four nearest
   Naively, only need 10,000 numbers (big
                     The model

   Divide high-res image into 7x7 patches
                       Our Model

   Divide high-res image into 7x7 patches
   Each patch is one node in the graph
        Or, each patch is one random variable in the
                 Another View

   Intuitively, we want to create a high-resolution
    image out of a collection of patches

                                         (Figure from Freeman)
   The compatibility between patches           is
    determined by how similar their borders are

    Patch 1         Patch 2
   First test, how well does it match the statistics of
    high-frequency, even if they're weird?
   Training Images:
        Revisiting The Issue of States
   In this algorithm, the state of each node is which
    candidate patch to use
   Tradeoff:
       Too many states – Maximizing P(h|l) is intractable
       Too few states – unable to represent high-res image
        well with a small number of discrete states.
   Solution:
       Use the observed low-resolution image to choose a
        small number of patches that are likely to represent
        the high-res image well
              Choosing Candidates
   Using a database
       Have to store database
       Have to search database
   Different Approach:
       Generate the candidates directly from the low-res
                         1-D Example

 Learn a set of interpolators
 Each interpolator creates a candidate high-resolution signal from the

low-resolution input
         Where does MS come from?
   Use a training database of high-resolution/low-
    resolution pairs to choose the best interpolators
   Find the interpolators using a clustering
    1. Cluster the pairs using k-means clustering
    2. For each cluster, find the interpolator that best
      predicts the high-resolution patches from the low-
      resolution patches
    3. For each pair, reassign it to the cluster whose
      interpolator best predicts the high-resolution patch
      from the low-resolution patch
    4. Repeat Steps 2 and 3 until convergence
How Many Interpolators are Needed?

   Can use the training set to estimate a lower bound
    on the error incurred using the interpolators
   For super-resolution, we used 64
   We use the image derivatives of neighboring
    patches to compute
                             Red Pixels – Border of Patch 1
                             Blue Pixels – Border of Patch 2
    Patch 1        Patch 2

                                             The difference
                                             a pair of adjacent
                                             red and blue pixels


   We can justify this choice in two ways
       Image statistics
       Image “sharpness”
              Image Statistics and

The distribution of the derivatives of a natural image is modeled
well by

where typically 0.7 <   < 1.2

Using                                 attempts to respect these
                     Image Sharpness

   Consider the problem of interpolating y1 from y0 and y2
   Model the distribution of y1 as
   If α > 1, then the most likely value of y1 is 1.5
       “Blurry” Edge
   If α < 1, then the most likely value of y1 is either 1 or 2
       Sharp edge
   α < 1 acts as a sharpness prior. It prefers sharp edges
                  Image Sharpness
   The compatibility function works the same way
   It favors setting the patches so that there are as
    few derivatives possible
   If there must be image derivatives, then
       If α > 1, then the compatibility function favors many
        small derivatives over a few large derivatives
       If α < 1, the compatibility function favors a few large
        derivatives over many small derivatives
   If α < 1 favors a high-res image with sharp edges
     Finding a high-resolution image
   Now that we have distribution, we can find a
    high-resolution image by maximizing

   One problem,
   With more than 2 states per node, maximizing
    is NP-Complete
   We're using 64 states per node!
   Maximizing P(h|l) is intractable, so we have to
    use an approximate technique
       Approximate technique – Not guaranteed to find the
        best value of h, but we hope it finds a pretty good
   Two Popular Techniques
    1.Loopy Belief Propagation (Pearl 88, Weiss 98,
      Freeman 2000)
    2.Graph Cuts (Boykov, Veksler, Zabih 2000)
         Loopy Belief Propagation
   Assume P(h|l) can be represented by a graph
    with no loops

   Can maximize P(h|l) using an algorithm based
    on passing messages between nodes in the graph
    (Pearl 88, Weiss 98)

   Messages encode belief about the state of each
                       Graph Cuts
   Start with an initial labeling of the graph
       Denote the probability of this labelling P(h0|l)
   How can we relabel the graph and increase
                      1    2     2    2

                      1    1     1    3

                      3    3     3    2

                      1    1     1    1
                        Graph Cuts
   Perform a swap:
       For two states s1 and s2, perform a swap by changing
        some nodes from s1 to s2 and vice-versa

         1    2     2    2             1    1    2    2

         1    1     1    3             1    1    1    3

         3    3     3    2             3    3    3    2

         1    1    1     1            1     2    2    1

   The optimal swap can be found in polynomial time
                   Graph Cuts
   Keep performing swaps between states until you
    can no longer increase P(h|l) with a single swap
   P(h|l) never decreases!
   Practically, convergence is guaranteed
                  Which should I use?
   Belief Propagation              Graph Cuts
       No guarantees about             No guarantees about
        maximizing P(h|l)                maximizing P(h|l)
       Not guaranteed to               Guaranteed to
        converge                         converge
       Can handle arbitrary            Not sure
        graphs                          Cannot produce
       Can estimate the                 marginals
        marginal probabilities          Tends to find better
        of P(h|l)                        solutions than BP
            caveat: these               (Tappen 2003)
             marginals will be
              Which did we use?
   Our model also includes a reconstruction
    constraint that forces the recovered high-
    resolution image to match the low-resolution
    image when it is down-sampled
   The peculiar graph structure led us to use belief
   (Already had BP code too)
   BP code available at

Actual High Res           Pixel Replicated      Bicubic Interpolation

       Greenspan et al.                 Our method

Actual High Res           Pixel Replicated      Bicubic Interpolation

       Greenspan et al.                 Our method
              CCD Demosaicing
   This approach is flexible enough to be applied to
    other image processing problems
   We also applied it to CCD Demosaicing
Common Artifacts

To top