A PDE Approach to Super-resolution with

Document Sample
A PDE Approach to Super-resolution with Powered By Docstoc
					                                                            2010 3rd International Congress on Image and Signal Processing (CISP2010)

                   A PDE Approach to Super-resolution with
                          Contrast Enhancement

                       Weili Zeng                                                            Xiaobo Lu
                School of Transportation                                             School of Automation
                  Southeast University                                                Southeast University
          Nanjing, Jiangsu 210096, P. R. China                                Nanjing, Jiangsu 210096, P. R. China
                zengwlj@yahoo.com.cn                                        Corresponding author: xblu2008@yahoo.cn

Abstract—Multi frame super resolution (SR) reconstruction            developed. Hardie et al.[11] proposed a joint MAP registration
algorithms make use of complimentary information among low           and restoration algorithm using a Gibbs image prior. Schultz
resolution (LR) images to yield a high resolution (HR) image. In     and Stevenson [2] proposed a Bayesian MAP estimator using a
this paper, we first present a fast partial differential equation    Huber-Markov random field model as an edge preserving prior.
(PDE) model for multi-frame image super resolution                   Farsiu et al [6] proposed an alternative data fidelity, or
reconstruction. We then combine our proposed super resolution        regularization term based on the L1 norm which has been
model with the local histogram equalization (LHE), which             shown to be robust to data outliers. More recently, X. L. Xue et
perform super resolution and enhance image contrast                  al. [12] proposed two new regularization items, termed as
simultaneously. It overcomes the shortcomings of recent
                                                                     locally adaptive bilateral total variation and consistency of
promising super resolution methods dealt with super resolution
and contrast enhancement separately. Our technique not only
                                                                     gradients, to keep edges and flat regions, which are implicitly
reconstructs a high-resolution image from several overlapping        described in LR images, sharp and smooth, respectively.
noisy low resolution images, but also enhances edges and image           In this paper, we first propose a fast partial differential
contrast while suppressing image noise during the reconstruction     equation (PDE) model for image super resolution. Moreover, it
process. Experiments show the effectiveness and practicability of    is well known that the classical super-resolution methods dealt
the proposed method and demonstrating its superiority to the         with super resolution and contrast enhancement separately. For
existing SR method.                                                  example, If contrast enhancement is applied firstly, the noise
    Keywords-Super-resolution; contrast enhancement; Partial
                                                                     may be enhanced significantly. On the other hand, if super-
differential equation (PDE) ; low resolution                         resolution is performed firstly, the weak edges may be
                                                                     destroyed. In both cases, the resulted images are hardly
                                                                     acceptable. Based on this reason, we combine our proposed
                      I.    INTRODUCTION                             super resolution model with the local histogram equalization,
    Super-resolution (SR) techniques aim at estimating a high-       which perform super-resolution and enhance image contrast
resolution (HR) image by utilizing complimentary information         simultaneously. It overcomes the shortcomings of recent
among low-resolution (LR) images. In many civilian and               promising super resolution methods dealt with super resolution
military applications, such as medical imaging, traffic video        and contrast enhancement separately. Experiments show the
sequences using low cost sensors, HR images are always               effectiveness and practicability of the proposed method.
required. Therefore, SR reconstruction is currently an active
                                                                         The structure of this paper is organized as follows. In
research topic in image processing [1-7].
                                                                     section II, we first describe the image degradation model, and
    SR reconstruction method can not only increase the image         we then derive a PDE method for super resolution that
pixels but also get more details and information by consider the     corresponds to the solution of the described image degradation
degradation process. There are many reasons leading to the           model. The numerical scheme for the PDE is followed in
degradation of the required video quality, such as atmospheric       section III. Experimental results are presented in Section 4 and
disturbances, motion, focus, down-sampling and the noise.            Section 5 gives some conclusions.
From a restoration point of way, the idea is to increase the
information content in the reference image by using the                                II.   PROPBLEM ALGORITHM
additional spatial-temporal information that is available in each
of the LR images. A variety of SR algorithms have been               A. Image Degradation Model
proposed since the multi-frame SR problem was first addressed
in [1]. Irani and Peleg [8] proposed an iterative back-projection        Given N consecutive low resolution images, denoted
method to address the super-resolution problem. Mann and             as Yk , k = 1, 2,..., n , which were cropped from a video,we
Picard [9] extended this to the projective case, and Zomet and       use the notation in [5] to formulate the general SR model. The
Peleg[10] rendered the original implementation more efficient,       relationship between the LR observation and the original HR
and applied it to image mosaics. To preserve edge information        image is expressed by a set of linear equations
while removing image noise, many algorithms have been

       978-1-4244-6516-3/10/$26.00 ©2010 IEEE                       600
                     Yk = Dk Bk Fk X + Vk , 1 ≤ k ≤ n                               (1)        Using the gradient descent method formulated as the time
                                                                                           evolution procedure to solve the Euler-Lagrange equation (6),
                                                                                           we have corresponding partial differential equation,
where X represents the HR image of size [ N1 N 2 × 1] which is
                                                                                           represented by
rearranged           in         lexicographic         order;    Yk and        Vk are
                                                                                                       ∂f     n
 M 1M 2 elements column representing the kth observed LR                                                  = −∑ Tk [Η k ( f ) − yk ] + λ div(        )             (7)
                                                                                                       ∂t    k =1                            | ∇f |
images and the additive noise, respectively; Bk denotes the
                                                                                           with homogeneous Neumann boundary conditions and
camera lens blur matrix, of size [ N1 N 2 × N1 N 2 ] ; Fk is a                             implementing a simple bilinear interpolation of the reference
[ N1 N 2 × N1 N 2 ] matrix that models the motion corresponding                            image as initialization.
to the reference frame; Dk is the decimation matrix , of size
                                                                                           C. Accelerating Evolution Procedure
[ M1M 2 × N1 N 2 ] .                                                                           The model (7) often converges very slowly to its steady
                                                                                           state since the parabolic term is quite singular for small
B. A PDE Method for Super-resolution                                                       gradients [14]. In order to accelerate the evolution procedure
    As we know, image SR problem is an ill-posed. In order to                              we multiply the Euler-Lagrange equation (6) by the magnitude
obtain a stable solution, a regularization term is always                                  of the gradient and our new time depend fast super resolution
introduced to the observation model. The following expression                              (FSR) model reads as follows
formulates a generalized minimization cost function in SR                                        ∂f             n
reconstruction process [10]                                                                         = − | ∇f | ∑ Tk [Η k ( f ) − yk ]+ | ∇f | λ div(        ) (8)
                                                                                                 ∂t            k =1                                  | ∇f |
                               ⎧n                                ⎫
             F ( X ) = arg min ⎨∑ ρ (Yk , Dk Bk Fk X ) + λΓ( X ) ⎬ .                (2)    with homogeneous Neumann boundary conditions and
                               ⎩ k =1                            ⎭                         implementing a simple bilinear interpolation of the reference
                                                                                           image as initialization as above.
where λ is the regularization parameter which provide a
tradeoff between the constrain regularized term Γ(⋅) and the                               D. Simultaneous Super-resolution and Contrast
residual data term ρ (⋅) .                                                                     Enhancement
    We rewrite the minimization cost function (2) by using an                                  We present a flow for simultaneous super resolution and
image representation instead of a matrix-vector representation.                            contrast enhancement. This is just an example of the possibility
It can be written, in a continuous form, as                                                of combining different algorithms in the same PDE.
                                                                                               Caselles et al. [15] proposed a PDE based approach to
                  ⎧ n                                                          ⎫
F ( f ) = arg min ⎨∫∫ ∑ ρ ( yk ( x, y), Η k ( f )( x, y))dxdy + λΓ( f ( x, y)) ⎬           perform global histogram equalization. However, because of
                  ⎩ k =1                                                       ⎭           the global modification not always produces good contrast, a
                                                                                    (3)    local histogram equalization algorithm which preserve the
                                                                                           family of level-sets of the image was proposed in [16]. Most
where Η k ( f ) is the transformed image of the super resolution                           recently, the authors in [17] generalized the PDE based
image f , with the image operation corresponding to Dk Bk Fk ,                             approach to contrast enhancement to adapt to any specified
                                                                                           stretch function for global and local contrast enhancement, of
and yk is the kth input image. The regularization term Γ(⋅) and                            the form
the residual data term ρ (⋅) are given by                                                               ∂f ( x, y, t )
                                                                                                                       = ϕ ( f ( x, y, t )) − f ( x, y, t ) ,     (9)
                          Γ( f ) = ∫∫ ∇( f ( x, y))dxdy                             (4)                      ∂t
                                                                                            where ϕ (⋅) is any stretch function specified by user. A widely
                                                                                            used stretch function is the local histogram equalization (LHE)
                                         1 n                                                [18], which will be used in our experiments.
      ∑ ρ(y ,Η
      k =1
                 k        k   ( f )) =     ∑ || yk ( x, y) − Η k ( f )( x, y) ||22 . (5)
                                         2 k =1                                                  As we know, one of the advantages of the use of PDE for
                                                                                            image processing is the possibility to combine algorithms, that
where ∇ denotes the gradient [13].                                                          is, if two procedures are given by
   The Euler-Lagrange equation associated to variational                                                  ∂f                         ∂f
problem (3) is                                                                                               = F1 ( f ( x, y, t )) ,    = F2 ( f ( x, y, t )) ,
                                                                                                          ∂t                         ∂t
                −∑Τk [Ηk ( f )(x, y) − yk (x, y)] + λdiv(       ) =0                (6)     then they can be combined as
                 k =1                                    | ∇f |
                                                                                                              = F1 ( f ( x, y, t )) + α F2 ( f ( x, y, t ))
where Τk is the transpose operator to Η k , div denotes the                                                ∂t

where F1 and F2 are two different operator,                            α ∈ R+ .
    Using the above property of PDE, the flows (8) and (9) can
be combined to obtain a new flow which performs super
resolution while improving the whole image contrast. The flow
is given by
             ∂f ( x, y, t )              N
                            = − | ∇f | ∑ Τk [Η k ( f ( x, y, t )) − yk ( x, y, t )]
                  ∂t                   k =1

                               + λ1 R1 ( ∇ f ) + λ 2 R 2 ( f )               (10)
where                                                                                                                               (a)                                        (b)

                           R1 (∇ f ) =| ∇ f | div (          ),
                                                      | ∇f |
                           R2 ( f ) = ϕ ( f ( x , y , t )) − f ( x , y , t ),
with homogeneous Neumann boundary conditions and
implementing a simple bilinear interpolation of the reference
image as initialization.

                                  III.      NUMERICAL SCHEME
    The PDE of (9) is not well defined at points where ∇f = 0 ,                                                                    (c)                                            (d)
due to the presence of the term 1/ ∇f . Then, it is common to                                               Figure 1.     (a) Original HR image. (b) One of six LR frames. (c) CSR. d)
slightly perturb the R1 (∇f ) to be                                                                                                             FSR

                                                                ∇f                                                        +    −
                                                                                                                       ( Dy + Dy ) fi ,nj
                                                                                                                                                                 +           +    −
                                                                                                                                                              ( Dx + Dx− )( Dy + Dy ) f i ,nj
                               R1 (∇f ) =| ∇f | div[                    ]                                        n
                                                                                                           [ f ] :=
                                                                                                                                            , [ f xy ]ij :=
                                                             | ∇ f | +ε                                         y ij
                                                                                                                              2h                                           4h 2
where       ε is a small positive parameter.
                                                                                                                            IV. EXPERIMENT RESULT
   Next we construct an explicit discrete scheme to
                                                                                                              In this section, we test our proposed method in both
numerically solve the derived PDE in equation (9). Let f ijn be
                                                                                                           simulated and real acquired LR sequences. For the simulated
the approximation to the value f ( xi , y j , tn ) , where                               xi = iΔx ,        experiment, the performance of the reconstruction algorithm
                                                                                                           was evaluated by measuring the peak signal to noise ratio
y j = j Δy and tn = nΔt , where Δx , Δy and Δt are the                                                     (PSNR) [17]. we will use Gaussian blur kernel, defined as
spatial step sizes and the time step size, respectively. Then our
first order scheme of (9) reads as follows:                                                                                                    1                 x2 + y2
                                                                                                                           h ( x, y ) =              exp(−               )                      (13)
        n           n −1
                                                                                                                                            2πα    2
                                                                                                                                                                  2α 2
     f −f
       ij          ij
                                                                                                          where α > 0 . In the following experiments, the criterion for
             Δt                                                                                           selecting the parameter in each experiment of each algorithm
                            ⎡ N                                                 ⎤                         is to choose parameters to produce visually most appealing
  = ( f xn ) 2 + ( f yn ) 2 ⎢ −∑ Tk [ yk − Η k ( f n )] + λ2 (ϕ ( f n ) − f n ) ⎥                         results [19].
                            ⎣ k =1                                              ⎦ ij
           ⎡ f n (( f n ) 2 + ε ) − 2 f xy f xn f yn + f yy (( f xn ) 2 + ε ) ⎤
                                           n              n
                                                                                                          A. Evaluation of Simulation Experiment
      + λ1 ⎢ xx y                                                             ⎥ . (11)                        We used a good quality image of size 256 × 256 pixels for
           ⎣                  (( f xn ) 2 + ( f yn )2 + ε )                   ⎥ ij
                                                                              ⎦                           the simulated data [Fig. 1(a)]. Each pixel has a value in [0,255].
To compute the right hand side of (10), we denote by                                                          In the first simulation experiment, we design to evaluate the
    ± n                       n           n                                                               fast convergence of our proposed method (denoted as “FSR”)
D f := ±( f
    x i, j                  i ±1, j   − f ) , D u := ±(u
                                         i, j
                                                    ± n
                                                    y i, j
                                                                     i , j ±1
                                                                                −u )
                                                                                  i, j                    compared with the super resolution (denoted as “CSR”)
                                                                                                          proposed in [13].
Then we have the following formulas to approximate the term
containing derivatives                                                                                        We first shifted this good quality image by two pixels in the
                                                    +  −
                                                                                                          vertical and horizontal direction. Then, this shifted image is
                Dx+ Dx−uin, j                      Dy Dy fi ,nj                   ( Dx+ + Dx− ) f i ,nj   corrupted by blurring it with 5 × 5 Gaussian blur kernel with
[ f xx ]ij :=                    , [ f yy ]ij :=
                                                                  [ f xn ]ij :=
                    h2                                 h2                                 2h              standard deviation equal to 6. Independent white Gaussian
                                                                                                          noise of variance 10 is was then added. Finally, the resulting

                                                                                                TABLE I.        COMPARISON PERFORMANCE
                                                                                                       FSR      FSR+LHE       LHE+FSR             FSRLHE
                                                                                       PSNR (dB)      22.8637     24.5508       24.3089           25.6703

Figure 2. PSNR values of interpolated images corresponding to 30 iterations
                                in Fig. 1.                                                             (a)                            (b)

image was down-sampled by a factor of 4 in each direction.
One of these degraded low-resolution images is shown in Fig.
1(b), in which the visual quality of this image is poor. We use
the method described in [20] to compute the motion vector.
The PNSR versus iteration numbers are shown Fig. 2. From the
plots of the corresponding PNSR values in Fig. 2 , we can get
that the our proposed FSR method needs smaller iteration
numbers but get better results than the CSR method.
The second simulation experiment is presented to verify the
good behavior of the simultaneous super resolution and
enhance image contrast method. Fig. 3(a) is the original high
                                                                                                (c)                                       (d)
resolution image. One of these degraded low-resolution images
is shown in Fig. 3(b) by decreasing brightness and contrast of
the six LR images generated in the previous experiment. Fig.
3(b) is the result using our proposed fast super resolution
without contrast enhancement (denoted as, ‘FSR’). Figure 3
(c)gives the result when we firstly performed histogram
equalization, and then followed by super resolution (denoted as
‘FSR+LHE’). If alternate the process order (denoted as,
‘LHE+FSR’), Fig. 4(d) is the result. The result of the
simultaneous super resolution and local histogram equalization
(denoted as, ‘FSR&LHE’) is given in Fig. 4(e). The PSNR
corresponding to these results are listed in Table I. It is obvious
that our proposed method outperforms the existing methods in                                    (e)                                         (f)
terms of both the quantitative measurement and visual
evaluation.                                                                         Figure 3. (a) Original HR image. (b) One of six LR frames. (c) FSR.
                                                                                                (d) FSR+LHE. (e) LHE+FSR. (f) FSR&LHE.
B. Evaluation of Real Data Experiment
    Here we use real surveillance sequences to validate the
effectiveness of the proposed super-resolution method. It is
assumed that the motion of the images during the sequence is a
globally translational motion.
The first surveillance sequence experiment shows a SR
reconstruction of a car sequence, which was obtained from a
surveillance video camera. One frame of this sequence is
shown in Fig. 4. Here the car is selected as our region of
interest. We used six LR images with one frame shown in Fig.
4(a) to obtain a resolution enhancement factor of 2. Fig. 5(b-e)
shows the result using FSR, SR+LHE, LHE+SR, and
SR&LHE, respectively. Evidently, our proposed method has
better visual quality.                                                                        Figure 4. Fig. 4. One LR frame in the car video

                                                                             Technologies R & D Program of China under grant


                                                                             [1]    T. S. Huang, R. Y. Tsai, “Multi-frame image restoration and
                                                                                    registration,” Adv. Comput. Vis. Image Process,Vol.1, 1984, 1: 317-339.
                                                                             [2]    R. Schultz, R. Stevenson, “Extraction of high-resolution frames from
                                                                                    video sequences,” IEEE Tran. Image Processing, Vol. 5, No. 6, 1996,
                                                                                    pp. 996-1011.
                  (a)                                  (b)                   [3]    M. Elad M , Y. Hel-Or, “A fast super-resolution reconstruction
                                                                                    algorithm for pure translational motion and common space invariant
                                                                                    blur.,” IEEE Tran. Image Processing, Vol. 10, 2001, pp.1187-1193.
                                                                             [4]    S. Farsiu, D. Robinson, M. Elad, et al., “Fast and robust multi-frame
                                                                                    super-resolution, ” IEEE Trans. Image Processing, Vol. 3. No. 10, 2004,
                                                                                    pp. 1327-1344.
                                                                             [5]    Takeda H, Milanfar P, Protter M and Elad M. Super-resolution without
                                                                                    explicit subpixel motion estimation. IEEE Tran. Image Processing, Vol.
                                                                                    18 No. 9, 2009, pp. 1958-1975.
                                                                             [6]    S. Farsiu, D. Robinson, M. Elad, et al., “Advances and challenges in
                                                                                    super-resolution,” Int. Journ. Imaging Sys. Technol., Vol. 14, No. 2,
                                                                                    2004, pp. 47-57..
                                                                             [7]    S. C. Park, M. K. Park et al., “Super-resolution image reconstruction: a
                   (c)                                 (d)                          technical review,” IEEE Signal Processing Magazine,Vol. 5, 2003, pp.
                                                                             [8]    M. Irani, S. Peleg, “Motion analysis for image enhancement: Resolution,
                                                                                    occlusion, and transparency,” J. Visual Commum. Image Represent,
                                                                                    Vol. 4, 1993, pp.324-335.
                                                                             [9]    S. Mann, R. W. Picard, “Virtual bellows: Constructing high quality stills
                                                                                    from video,” Proc. IEEE Int. Conf. Image Process, 1994.
                                                                             [10]   A. Zomet, S. Peleg, “Efficient super-resolution and applications to
                                                                                    mosaics,” Proc. Intl. Conf. Patt. Recog., 2000, 579-583.
                                                                             [11]   R. Hardie, K. Barnard, E. Armstrong, “Joint MAP registration and high
                                                                                    resolution image estimation using a sequence of under-sampled images,”
                                                                                    IEEE Transactions on Image Processing, Vol. 6, No. 12, 1997, pp. 1621-
                                   (e)                                       [12]   X. L. Li, Y. T. Hu, et al., “A multi-frame image super-resolution
                                                                                    method,” Signal Processing, Vol. 90, 2010, pp. 405-414.
 Figure 5. SR reconstruction results from real LR video sequence. (a) One    [13]   A. Marquina, S. J. Osher, “Image super-resolution by TV Regularization
    frame of the LR images. (b) FSR. (c) FSR+LHE. (d) LHE+FSR. (e)                  and bregman iteration,” Journal of Scientific and Computing, Vol. 37,
                               FSR&LHE.                                             2008, pp. 367-382.
                                                                             [14]   A. Marquina, S. Osher, “Explicit algorithms for a new time dependent
                                                                                    model based on level set motion for nonlinear deblurring and noise
                         V.    CONCLUSION                                           removal,” SIAM:J. SCi. Comput, Vol. 22, No. 1, 2000, pp. 387-405.
                                                                             [15]   G. Sapiro, V. Caselles, “Contrast enhancement via image evolution
    In this paper, we propose a PDE approach to super                               flows,” Graphical Models and Image Processing, Vol. 59, No. 6, 1997,
resolution with contrast enhancement. The two main features of                      pp. 407-416.
this paper are the following: On the one hand, we propose a                  [16]   V. Caselles, J. L. Lisani, J. M. Morel, et al., “Shape preserving local
fast partial differential equation (PDE) model for multi-frame                      histogram modification,” IEEE Transactions on Image Processing, Vol.
super resolution reconstruction. One the other hand, we                             8, No. 2, 1999, pp. 220-230.
combine our proposed super resolution model with the local                   [17]   J. Ke, Y. Q. Hou, et al., “An improved algorithm for shape preserving
histogram equalization, which perform super-resolution and                          contrast enhancement,” Acta Photonica Sinica, Vol. 38, No. 1, 2009, pp.
enhance image contrast and simultaneously. It overcomes the
                                                                             [18]   Q. Huynh-Thu, M. Ghanhari, “Scope of validity of PSNR in
shortcomings of recent promising super resolution methods                           image/video quality assessment,” Electronics Letters, Vol. 44, No. 13,
dealt with super resolution and contrast enhancement                                2008, pp. 800-801.
separately. Experiment results show the good behavior of the                 [19]   L. P. Zhang, H. Y. Zhang, et al., “A super-resolution reconstruction
proposed method.                                                                    algorithm for surveillance images, ” Signal Processing, Vol. 90, 2010,
                                                                                    pp. 848-859.
                         ACKNOWLEDGMENT                                      [20]   G. Rochefort, F. Champagnat, et al., “Animproved observation model
                                                                                    for super-resolution under affine motion,” .IEEE Transactionson Image
   This work was supported by National Natural Science                              Processing, Vol. 15, No. 11, 2006, pp.3325-3337.
Foundation of China under grant 60972001 and National Key


Description: A MODIFIED UNSHARP-MASKING TECHNIQUE FOR IMAGE CONTRAST, A NEW APPROACH FOR VERY DARK VIDEO DENOISING AND ENHANCEMENT, A PDE Approach to Super-resolution with, A Unified Histogram and Laplacian Based for Image, An Adaptive Image Enhancement Technique, An Improved Retinex Image Enhancement Technique, Automatic Exact Histogram Specification for, Bayesian Foreground and Shadow Detection in, Color Image Enhancement and Denoising Using an Optimized Filternet, Content Based Image Retrieval Using, Contrast Enhancement for Ziehl-Neelsen Tissue, DETAIL WARPING BASED VIDEO SUPER-RESOLUTION USING IMAGE GUIDES, Gray-level Image Enhancement By Particle Swarm, Image Contrast Enhancement based on Histogram, Image Enhancement and Segmentation Using Dark, Image Enhancement Technique Based On Improved, Image Quality Improvement fo r Electrophoretic Displays by, Image Reconstruction Using Particle Filters and, IMPROVED IDENTIFICATION OF IRIS AND EYELASH FEATURES, Improving Colour Image Segmentation on Acute, K�R ANALİZ Y�NTEMLERİ İLE İMGE İYİLEŞTİRME, NATURAL RENDERING OF COLOR IMAGE BASED ON RETINEX, Power-Constrained Contrast Enhancement, Research on Road Image Fusion Enhancement, Shadow Detection and Compensation in High Resolution Satellite Image Based, Smoothing Cephalographs Using Modified, Three-Dimensional Computational Integral Imaging Reconstruction by Use of, Towards integrating level-3 Features with