Docstoc

Super-Resolution

Document Sample
Super-Resolution Powered By Docstoc
					Super-Resolution

   Barak Zackay
   Yaron Kassner
                    Outline
• Introduction to Super-Resolution
• Reconstruction Based Super Resolution
  – An Algorithm
  – Limits on Reconstruction Based Super Resolution
• Example Based Super Resolution
  – Halucination
  – Example Based
  – Single Image Super Resolution
• Summary
 Introduction to
Super Resolution
    Definition of the Problem
• Super-resolution is the process of combining
  multiple low resolution images to form a higher
  resolution one.
• No cheating!
  – Resulting image should represent reality better than
    all the input images.
The Imaging Process
          Physical Properties
• Each camera suffers from some inherent optical
  issues:
  – Finite size of the aperture - generates blur, modeled
    by the Point-Spread-Function (PSF).
  – Noise
         Mathematical Model
• Each pixel in the resulting image is given by:




• Loi(m) – the i-th LR image in
  pixel m.
• Ei (x) – total photon count
  from the direction x
• PSFi – Point Spread Function
            Deresolution
• Given HR image
• Project to LR image
• Each LR pixel is a
  linear combination         LR
  of HR pixels


                        HR    HR   HR
      Reconstruction-based
        Super Resolution
• Reconstruct hidden HR pixels out of
  known linear combinations.



                       LR         LR




        HR   HR   HR    HR
                       LR    HR    HR
                                  LR    HR
            Example-Based
            Super Resolution
• Use prior knowledge to reconstruct a HR
  image.




 Prior Knowledge of faces
Reconstruction Based
  Super Resolution
                   from
Improving Resolution by Image Registration

     Michal Irani and Shmuel Peleg
             Basic Idea
• The HR image should create the LR
  images when deresoluted.
                       Notation
•    gk :     The kth observed LR image.
•    f  n : The approximation to the HR image after n
     iterations.
•        
     g : The LR image obtained by applying the
     k
         n



     simulated imaging process to f n.
•   h PSF : The point spread function of the imaging blur.
•    x : a HR pixel

•    y : a LR pixel influenced by x

•   z y : The center of the receptive field of y.
       Problem Formulation
• Find a HR image   f n    , that gives   g n   g   .

  –
           Algorithm Overview
• Register the LR images.
• Guess the HR image f   .      0
                                      n




• Iteration n:
                                                               n 
  – Simulate the imaging process to create                gk

    from f n .
                         
  – Compare         gk
                         n
                            and g .
                             k


  – Correct      f  n  in the direction of the error.

• output    f n 
     Registration
LR        LR
                         LR         LR


LR        LR
                         LR         LR




                  LR          LR



     HR   HR       HR
               HR LR        HR
                        HR LR      HR
                              Iteration
• Take the current guess.
                                     n 
• Decrease its resolution to get g k
• Update each HR pixel x according to the error in all LR pixels (y) it
  influences.c is a constant normalizing factor.




    – c is a constant normalizing factor.
    – Yk,x is the group of all pixels y that are influenced by x.

    – hxy BP is a back-projection kernel applied on x  z y that represents the
      way the HR pixel x should be updated from y.
                            Wasach
One of three input images




Initial guess (average of input images)




Output
                 Debluring




Original Image     Blurred Image   Restored Image
                Wasach




Blurred Image    Initial Guess   Restored Image
Limits on Reconstruction
     Based Methods
                  from
 Limits on Super-Resolution and How to
              Break Them
      Simon Baker and Takeo Kanade
 Large Magnification Factor is
        Problematic
• Large magnification factor causes:
  – Overly smooth HR image
  – Fine details are not recovered

• An explanation is needed.
             Evil Example
• Suppose we want to increase the
  resolution by exactly M=2.
• Lets look on a checkboard like scene,
  where each pixel is either white or black.


                              LR



                         HR   HR   HR
Information is Inherently Missing
• The resulting image would be grey
  independently from the offset of the LR
  image!
• Conclusion: some information is inherently
  missing on our LR images!
When M is not an Integer




     LR



          HR   HR   HR
        Limits of Super-Resolution
• Size of LR images: N pixels.
• Size of HR image: NM 2 pixels.
• Each HR pixel can be added noise of amplitude smaller
  than M 2 which wont change the LR image!
• Volume of possible HR solutions: O(M 2N) 1
• It can be shown that under practical considerations the
  effective magnification factor (M) is bounded by 1.6, no
  matter how many LR images are taken2.

1 Limits on Super-Resolution and How to Break Them, Simon Baker and Takeo Kanade
2 Fundamental Limits of Reconstruction-Based Superresolution Algorithms under Local Translation,
    Zhouchen Lin, and Heung-Yeung Shum
                     Break
• Introduction to Super-Resolution
• Reconstruction Based Super Resolution
  – An Algorithm
  – Limits on Reconstruction Based Super Resolution
• Example Based Super Resolution
  – Halucination
  – Example Based
  – Single Image Super Resolution
• Summary
Example Based
Super Resolution
 Introduction to Example-Based
        Super Resolution
• Reconstruction constraints are not
  enough.
• One has to use prior knowledge of the
  image to break the reconstruction limits.
• The following algorithms will use priors
  learned from databases of example
  images.
   Recogstruction or
     Hallucination
                      from
Limits on Super-Resolution and How to Break Them

        Simon Baker and Takeo Kanade
              General Idea
• Find a HR image Su that satisfies two
  kinds of constraints:
  – Reconstruction constraints: When projected to
    the LR dimensions, the image is similar to the
    observed input images.
  – Recognition constraints: The pixels of Su
    should resemble pixels from images in the DB
    that where found to have similar features to
    the observed LR images’ features.
           MAP formulation
• To solve the problem, given the LR
  images, we need to find the HR image
  that maximizes



    - Su: the HR image    Reconstruction Constraints
    - Lo: the LR images
                                          Recognition Constraints
    Reconstruction Constraints
• The probability of the LR images given the HR
  image can be computed from the distance
  between the deresoluted HR image and the LR
  images.




  –        : the noise variance
  – PSF: Point Spread Function
  – ri z : The pixel in Lo that corresponds to pixel z in Su.
  – m: a LR pixel index
     Recognition: LR features
• We use “Parent Structures” to describe LR
  features.
       Recognition: Choosing the
          Pixels from the DB




PS = Parent Structure
F = Features – like First deriviative, or Laplacian
             Formulation of Recognition
                    Constraints
• Instead of estimating the probability of the
  HR image, Su, we estimate its probability
  given each pixel’s “recognition”.



H0 – Horizontal derivative
V0 – Vertical derivative.
    - Variance of the recognition errors.
T - the training images.
BI – best images for the pixels of the LR images.
BP – best pixel indices in the best images for the pixels of the LR images.
Ci,BP,BI – Class of all images that would have the Best corresponding Images BI,
and the Best corresponding Pixels BP in the db.
      - The function that fits a LR pixel index to the corresponding HR pixel index.
2k – the ratio between the HR image scale and the LR image scale.
           Maximization

• Note that the function we need to
  maximize is quadratic with the HR image’s
  pixels.




• Do gradient descent.
          Algorithm Summary
• Preliminary work:
  – Take a training set of images.
  – Build a DB that matches parent structures to HR
    pixels.
• Compute the reconstruction constraints.
• For each LR image:
  – For each HR pixel index:
     • Search for the corresponding parent structure in the DB.
• Find the HR image that fits best both the
  reconstruction constraints and the HR pixels
  extracted from the database.
Comparison
Comparison
Best and Worst Image
Noise Effect
Image Size
Hallucination
Hallucination
Results on Text
   Example Based
   Super Resolution

William T. Freeman, Thouis R. Jones
        and Egon C. Pasztor
          Algorithm Overview
• Construct a DB of matching LR-HR patches

• Algorithmically find the most coherent patch
  assignment to generate a good image
         Constructing the DB
• Given a DB of images
• Make a table from LR patches to HR patches.
• Each image in the DB is treated as follows:
  – Take each 7x7 patch from the image and deresolute
    into a 5x5 patch
  – Normalize the 5x5 patches to have the same mean
    and relative contrast.
  – Arrange the DB by the low frequencies of the LR
    patches
      Local Patch Matching
• Match a LR patch to a HR patch from the DB,
  using low frequencies.
• Get an estimation to the unknown (high)
  frequencies, based on the match.
• Remaining problem: match between neighboring
  overlapping patches.
      Global Patch Matching
• Run over patches from left to right and from top
  to bottom
• Match each patch its nearest neighbor in the DB
  using the predetermined patches as additional
  constraints.
Algorithm
Wasach
Wasach
               Wasach




Cubic-spline   Super-resolution   True high-resolution image
Wasach
Complete Failure
Priors are definitely used!
Super Resolution From a
     Single Image

 Daniel Glaser, Shai Bagon and
          Michal Irani
Patch Redundancy in a Single
          Image
Employing in-scale Patch
     Redundancy
   Employing Cross-scale Patch
          Redundancy
• Build a cascade of decreasing resolution
  images from the LR image.
• For each patch in the LR image, search for
  its Nearest Neighbour in the even lower
  resolution image.
• Take the found neighbour’s parent in the
  original LR image and copy it to be the HR
  image.
Combining the Methods
                           Wasach




Bicubic interpolation   Unified single-image SR (x3)   Ground truth image
                        Wasach




Bicubic interpolation       Unified single-image SR (x3)
Wasach
Wasach
Wasach
Wasach
Wasach
Wasach
Wasach
Wasach
Wasach
                     Summary
• We have presented two basic approaches for super
  resolution:
   – Reconstruction-based – which simply tries to reverse the
     imaging process
   – Example-based – which uses example images to reconstruct the
     original image.
• We have shown that there are limits to reconstruction
  based methods, which are independent of the number of
  LR images we use.
• We have presented an algorithm that combines both
  approaches to achieve SR from a single image.
Questions?

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:14
posted:6/2/2012
language:English
pages:77