2010 3rd International Congress on Image and Signal Processing (CISP2010)
A PDE Approach to Super-resolution with
Weili Zeng Xiaobo Lu
School of Transportation School of Automation
Southeast University Southeast University
Nanjing, Jiangsu 210096, P. R. China Nanjing, Jiangsu 210096, P. R. China
email@example.com Corresponding author: firstname.lastname@example.org
Abstract—Multi frame super resolution (SR) reconstruction developed. Hardie et al. proposed a joint MAP registration
algorithms make use of complimentary information among low and restoration algorithm using a Gibbs image prior. Schultz
resolution (LR) images to yield a high resolution (HR) image. In and Stevenson  proposed a Bayesian MAP estimator using a
this paper, we first present a fast partial differential equation Huber-Markov random field model as an edge preserving prior.
(PDE) model for multi-frame image super resolution Farsiu et al  proposed an alternative data fidelity, or
reconstruction. We then combine our proposed super resolution regularization term based on the L1 norm which has been
model with the local histogram equalization (LHE), which shown to be robust to data outliers. More recently, X. L. Xue et
perform super resolution and enhance image contrast al.  proposed two new regularization items, termed as
simultaneously. It overcomes the shortcomings of recent
locally adaptive bilateral total variation and consistency of
promising super resolution methods dealt with super resolution
and contrast enhancement separately. Our technique not only
gradients, to keep edges and flat regions, which are implicitly
reconstructs a high-resolution image from several overlapping described in LR images, sharp and smooth, respectively.
noisy low resolution images, but also enhances edges and image In this paper, we first propose a fast partial differential
contrast while suppressing image noise during the reconstruction equation (PDE) model for image super resolution. Moreover, it
process. Experiments show the effectiveness and practicability of is well known that the classical super-resolution methods dealt
the proposed method and demonstrating its superiority to the with super resolution and contrast enhancement separately. For
existing SR method. example, If contrast enhancement is applied firstly, the noise
Keywords-Super-resolution; contrast enhancement; Partial
may be enhanced significantly. On the other hand, if super-
differential equation (PDE) ; low resolution resolution is performed firstly, the weak edges may be
destroyed. In both cases, the resulted images are hardly
acceptable. Based on this reason, we combine our proposed
I. INTRODUCTION super resolution model with the local histogram equalization,
Super-resolution (SR) techniques aim at estimating a high- which perform super-resolution and enhance image contrast
resolution (HR) image by utilizing complimentary information simultaneously. It overcomes the shortcomings of recent
among low-resolution (LR) images. In many civilian and promising super resolution methods dealt with super resolution
military applications, such as medical imaging, traffic video and contrast enhancement separately. Experiments show the
sequences using low cost sensors, HR images are always effectiveness and practicability of the proposed method.
required. Therefore, SR reconstruction is currently an active
The structure of this paper is organized as follows. In
research topic in image processing [1-7].
section II, we first describe the image degradation model, and
SR reconstruction method can not only increase the image we then derive a PDE method for super resolution that
pixels but also get more details and information by consider the corresponds to the solution of the described image degradation
degradation process. There are many reasons leading to the model. The numerical scheme for the PDE is followed in
degradation of the required video quality, such as atmospheric section III. Experimental results are presented in Section 4 and
disturbances, motion, focus, down-sampling and the noise. Section 5 gives some conclusions.
From a restoration point of way, the idea is to increase the
information content in the reference image by using the II. PROPBLEM ALGORITHM
additional spatial-temporal information that is available in each
of the LR images. A variety of SR algorithms have been A. Image Degradation Model
proposed since the multi-frame SR problem was first addressed
in . Irani and Peleg  proposed an iterative back-projection Given N consecutive low resolution images, denoted
method to address the super-resolution problem. Mann and as Yk , k = 1, 2,..., n , which were cropped from a video，we
Picard  extended this to the projective case, and Zomet and use the notation in  to formulate the general SR model. The
Peleg rendered the original implementation more efficient, relationship between the LR observation and the original HR
and applied it to image mosaics. To preserve edge information image is expressed by a set of linear equations
while removing image noise, many algorithms have been
978-1-4244-6516-3/10/$26.00 ©2010 IEEE 600
Yk = Dk Bk Fk X + Vk , 1 ≤ k ≤ n (1) Using the gradient descent method formulated as the time
evolution procedure to solve the Euler-Lagrange equation (6),
we have corresponding partial differential equation,
where X represents the HR image of size [ N1 N 2 × 1] which is
rearranged in lexicographic order; Yk and Vk are
M 1M 2 elements column representing the kth observed LR = −∑ Tk [Η k ( f ) − yk ] + λ div( ) (7)
∂t k =1 | ∇f |
images and the additive noise, respectively; Bk denotes the
with homogeneous Neumann boundary conditions and
camera lens blur matrix, of size [ N1 N 2 × N1 N 2 ] ; Fk is a implementing a simple bilinear interpolation of the reference
[ N1 N 2 × N1 N 2 ] matrix that models the motion corresponding image as initialization.
to the reference frame; Dk is the decimation matrix , of size
C. Accelerating Evolution Procedure
[ M1M 2 × N1 N 2 ] . The model (7) often converges very slowly to its steady
state since the parabolic term is quite singular for small
B. A PDE Method for Super-resolution gradients . In order to accelerate the evolution procedure
As we know, image SR problem is an ill-posed. In order to we multiply the Euler-Lagrange equation (6) by the magnitude
obtain a stable solution, a regularization term is always of the gradient and our new time depend fast super resolution
introduced to the observation model. The following expression (FSR) model reads as follows
formulates a generalized minimization cost function in SR ∂f n
reconstruction process  = − | ∇f | ∑ Tk [Η k ( f ) − yk ]+ | ∇f | λ div( ) (8)
∂t k =1 | ∇f |
F ( X ) = arg min ⎨∑ ρ (Yk , Dk Bk Fk X ) + λΓ( X ) ⎬ . (2) with homogeneous Neumann boundary conditions and
⎩ k =1 ⎭ implementing a simple bilinear interpolation of the reference
image as initialization as above.
where λ is the regularization parameter which provide a
tradeoff between the constrain regularized term Γ(⋅) and the D. Simultaneous Super-resolution and Contrast
residual data term ρ (⋅) . Enhancement
We rewrite the minimization cost function (2) by using an We present a flow for simultaneous super resolution and
image representation instead of a matrix-vector representation. contrast enhancement. This is just an example of the possibility
It can be written, in a continuous form, as of combining different algorithms in the same PDE.
Caselles et al.  proposed a PDE based approach to
⎧ n ⎫
F ( f ) = arg min ⎨∫∫ ∑ ρ ( yk ( x, y), Η k ( f )( x, y))dxdy + λΓ( f ( x, y)) ⎬ perform global histogram equalization. However, because of
⎩ k =1 ⎭ the global modification not always produces good contrast, a
(3) local histogram equalization algorithm which preserve the
family of level-sets of the image was proposed in . Most
where Η k ( f ) is the transformed image of the super resolution recently, the authors in  generalized the PDE based
image f , with the image operation corresponding to Dk Bk Fk , approach to contrast enhancement to adapt to any specified
stretch function for global and local contrast enhancement, of
and yk is the kth input image. The regularization term Γ(⋅) and the form
the residual data term ρ (⋅) are given by ∂f ( x, y, t )
= ϕ ( f ( x, y, t )) − f ( x, y, t ) , (9)
Γ( f ) = ∫∫ ∇( f ( x, y))dxdy (4) ∂t
where ϕ (⋅) is any stretch function specified by user. A widely
used stretch function is the local histogram equalization (LHE)
1 n , which will be used in our experiments.
∑ ρ(y ,Η
k k ( f )) = ∑ || yk ( x, y) − Η k ( f )( x, y) ||22 . (5)
2 k =1 As we know, one of the advantages of the use of PDE for
image processing is the possibility to combine algorithms, that
where ∇ denotes the gradient . is, if two procedures are given by
The Euler-Lagrange equation associated to variational ∂f ∂f
problem (3) is = F1 ( f ( x, y, t )) , = F2 ( f ( x, y, t )) ,
−∑Τk [Ηk ( f )(x, y) − yk (x, y)] + λdiv( ) =0 (6) then they can be combined as
k =1 | ∇f |
= F1 ( f ( x, y, t )) + α F2 ( f ( x, y, t ))
where Τk is the transpose operator to Η k , div denotes the ∂t
where F1 and F2 are two different operator, α ∈ R+ .
Using the above property of PDE, the flows (8) and (9) can
be combined to obtain a new flow which performs super
resolution while improving the whole image contrast. The flow
is given by
∂f ( x, y, t ) N
= − | ∇f | ∑ Τk [Η k ( f ( x, y, t )) − yk ( x, y, t )]
∂t k =1
+ λ1 R1 ( ∇ f ) + λ 2 R 2 ( f ) (10)
where (a) (b)
R1 (∇ f ) =| ∇ f | div ( ),
| ∇f |
R2 ( f ) = ϕ ( f ( x , y , t )) − f ( x , y , t ),
with homogeneous Neumann boundary conditions and
implementing a simple bilinear interpolation of the reference
image as initialization.
III. NUMERICAL SCHEME
The PDE of (9) is not well defined at points where ∇f = 0 , (c) (d)
due to the presence of the term 1/ ∇f . Then, it is common to Figure 1. (a) Original HR image. (b) One of six LR frames. (c) CSR. d)
slightly perturb the R1 (∇f ) to be FSR
∇f + −
( Dy + Dy ) fi ,nj
+ + −
( Dx + Dx− )( Dy + Dy ) f i ,nj
R1 (∇f ) =| ∇f | div[ ] n
[ f ] :=
, [ f xy ]ij :=
| ∇ f | +ε y ij
2h 4h 2
where ε is a small positive parameter.
IV. EXPERIMENT RESULT
Next we construct an explicit discrete scheme to
In this section, we test our proposed method in both
numerically solve the derived PDE in equation (9). Let f ijn be
simulated and real acquired LR sequences. For the simulated
the approximation to the value f ( xi , y j , tn ) , where xi = iΔx , experiment, the performance of the reconstruction algorithm
was evaluated by measuring the peak signal to noise ratio
y j = j Δy and tn = nΔt , where Δx , Δy and Δt are the (PSNR) . we will use Gaussian blur kernel, defined as
spatial step sizes and the time step size, respectively. Then our
first order scheme of (9) reads as follows: 1 x2 + y2
h ( x, y ) = exp(− ) (13)
n n −1
where α > 0 . In the following experiments, the criterion for
Δt selecting the parameter in each experiment of each algorithm
⎡ N ⎤ is to choose parameters to produce visually most appealing
= ( f xn ) 2 + ( f yn ) 2 ⎢ −∑ Tk [ yk − Η k ( f n )] + λ2 (ϕ ( f n ) − f n ) ⎥ results .
⎣ k =1 ⎦ ij
⎡ f n (( f n ) 2 + ε ) − 2 f xy f xn f yn + f yy (( f xn ) 2 + ε ) ⎤
A. Evaluation of Simulation Experiment
+ λ1 ⎢ xx y ⎥ . (11) We used a good quality image of size 256 × 256 pixels for
⎣ (( f xn ) 2 + ( f yn )2 + ε ) ⎥ ij
⎦ the simulated data [Fig. 1(a)]. Each pixel has a value in [0,255].
To compute the right hand side of (10), we denote by In the first simulation experiment, we design to evaluate the
± n n n fast convergence of our proposed method (denoted as “FSR”)
D f := ±( f
x i, j i ±1, j − f ) , D u := ±(u
y i, j
i , j ±1
i, j compared with the super resolution (denoted as “CSR”)
proposed in .
Then we have the following formulas to approximate the term
containing derivatives We first shifted this good quality image by two pixels in the
vertical and horizontal direction. Then, this shifted image is
Dx+ Dx−uin, j Dy Dy fi ,nj ( Dx+ + Dx− ) f i ,nj corrupted by blurring it with 5 × 5 Gaussian blur kernel with
[ f xx ]ij := , [ f yy ]ij :=
[ f xn ]ij :=
h2 h2 2h standard deviation equal to 6. Independent white Gaussian
noise of variance 10 is was then added. Finally, the resulting
TABLE I. COMPARISON PERFORMANCE
FSR FSR+LHE LHE+FSR FSRLHE
PSNR (dB) 22.8637 24.5508 24.3089 25.6703
Figure 2. PSNR values of interpolated images corresponding to 30 iterations
in Fig. 1. (a) (b)
image was down-sampled by a factor of 4 in each direction.
One of these degraded low-resolution images is shown in Fig.
1(b), in which the visual quality of this image is poor. We use
the method described in  to compute the motion vector.
The PNSR versus iteration numbers are shown Fig. 2. From the
plots of the corresponding PNSR values in Fig. 2 , we can get
that the our proposed FSR method needs smaller iteration
numbers but get better results than the CSR method.
The second simulation experiment is presented to verify the
good behavior of the simultaneous super resolution and
enhance image contrast method. Fig. 3(a) is the original high
resolution image. One of these degraded low-resolution images
is shown in Fig. 3(b) by decreasing brightness and contrast of
the six LR images generated in the previous experiment. Fig.
3(b) is the result using our proposed fast super resolution
without contrast enhancement (denoted as, ‘FSR’). Figure 3
(c)gives the result when we firstly performed histogram
equalization, and then followed by super resolution (denoted as
‘FSR+LHE’). If alternate the process order (denoted as,
‘LHE+FSR’), Fig. 4(d) is the result. The result of the
simultaneous super resolution and local histogram equalization
(denoted as, ‘FSR&LHE’) is given in Fig. 4(e). The PSNR
corresponding to these results are listed in Table I. It is obvious
that our proposed method outperforms the existing methods in (e) (f)
terms of both the quantitative measurement and visual
evaluation. Figure 3. (a) Original HR image. (b) One of six LR frames. (c) FSR.
(d) FSR+LHE. (e) LHE+FSR. (f) FSR&LHE.
B. Evaluation of Real Data Experiment
Here we use real surveillance sequences to validate the
effectiveness of the proposed super-resolution method. It is
assumed that the motion of the images during the sequence is a
globally translational motion.
The first surveillance sequence experiment shows a SR
reconstruction of a car sequence, which was obtained from a
surveillance video camera. One frame of this sequence is
shown in Fig. 4. Here the car is selected as our region of
interest. We used six LR images with one frame shown in Fig.
4(a) to obtain a resolution enhancement factor of 2. Fig. 5(b-e)
shows the result using FSR, SR+LHE, LHE+SR, and
SR&LHE, respectively. Evidently, our proposed method has
better visual quality. Figure 4. Fig. 4. One LR frame in the car video
Technologies R & D Program of China under grant
 T. S. Huang, R. Y. Tsai, “Multi-frame image restoration and
registration,” Adv. Comput. Vis. Image Process,Vol.1, 1984, 1: 317-339.
 R. Schultz, R. Stevenson, “Extraction of high-resolution frames from
video sequences,” IEEE Tran. Image Processing, Vol. 5, No. 6, 1996,
(a) (b)  M. Elad M , Y. Hel-Or, “A fast super-resolution reconstruction
algorithm for pure translational motion and common space invariant
blur.,” IEEE Tran. Image Processing, Vol. 10, 2001, pp.1187-1193.
 S. Farsiu, D. Robinson, M. Elad, et al., “Fast and robust multi-frame
super-resolution, ” IEEE Trans. Image Processing, Vol. 3. No. 10, 2004,
 Takeda H, Milanfar P, Protter M and Elad M. Super-resolution without
explicit subpixel motion estimation. IEEE Tran. Image Processing, Vol.
18 No. 9, 2009, pp. 1958-1975.
 S. Farsiu, D. Robinson, M. Elad, et al., “Advances and challenges in
super-resolution,” Int. Journ. Imaging Sys. Technol., Vol. 14, No. 2,
2004, pp. 47-57..
 S. C. Park, M. K. Park et al., “Super-resolution image reconstruction: a
(c) (d) technical review,” IEEE Signal Processing Magazine,Vol. 5, 2003, pp.
 M. Irani, S. Peleg, “Motion analysis for image enhancement: Resolution,
occlusion, and transparency,” J. Visual Commum. Image Represent,
Vol. 4, 1993, pp.324-335.
 S. Mann, R. W. Picard, “Virtual bellows: Constructing high quality stills
from video,” Proc. IEEE Int. Conf. Image Process, 1994.
 A. Zomet, S. Peleg, “Efficient super-resolution and applications to
mosaics,” Proc. Intl. Conf. Patt. Recog., 2000, 579-583.
 R. Hardie, K. Barnard, E. Armstrong, “Joint MAP registration and high
resolution image estimation using a sequence of under-sampled images,”
IEEE Transactions on Image Processing, Vol. 6, No. 12, 1997, pp. 1621-
(e)  X. L. Li, Y. T. Hu, et al., “A multi-frame image super-resolution
method,” Signal Processing, Vol. 90, 2010, pp. 405-414.
Figure 5. SR reconstruction results from real LR video sequence. (a) One  A. Marquina, S. J. Osher, “Image super-resolution by TV Regularization
frame of the LR images. (b) FSR. (c) FSR+LHE. (d) LHE+FSR. (e) and bregman iteration,” Journal of Scientific and Computing, Vol. 37,
FSR&LHE. 2008, pp. 367-382.
 A. Marquina, S. Osher, “Explicit algorithms for a new time dependent
model based on level set motion for nonlinear deblurring and noise
V. CONCLUSION removal,” SIAM:J. SCi. Comput, Vol. 22, No. 1, 2000, pp. 387-405.
 G. Sapiro, V. Caselles, “Contrast enhancement via image evolution
In this paper, we propose a PDE approach to super flows,” Graphical Models and Image Processing, Vol. 59, No. 6, 1997,
resolution with contrast enhancement. The two main features of pp. 407-416.
this paper are the following: On the one hand, we propose a  V. Caselles, J. L. Lisani, J. M. Morel, et al., “Shape preserving local
fast partial differential equation (PDE) model for multi-frame histogram modification,” IEEE Transactions on Image Processing, Vol.
super resolution reconstruction. One the other hand, we 8, No. 2, 1999, pp. 220-230.
combine our proposed super resolution model with the local  J. Ke, Y. Q. Hou, et al., “An improved algorithm for shape preserving
histogram equalization, which perform super-resolution and contrast enhancement,” Acta Photonica Sinica, Vol. 38, No. 1, 2009, pp.
enhance image contrast and simultaneously. It overcomes the
 Q. Huynh-Thu, M. Ghanhari, “Scope of validity of PSNR in
shortcomings of recent promising super resolution methods image/video quality assessment,” Electronics Letters, Vol. 44, No. 13,
dealt with super resolution and contrast enhancement 2008, pp. 800-801.
separately. Experiment results show the good behavior of the  L. P. Zhang, H. Y. Zhang, et al., “A super-resolution reconstruction
proposed method. algorithm for surveillance images, ” Signal Processing, Vol. 90, 2010,
ACKNOWLEDGMENT  G. Rochefort, F. Champagnat, et al., “Animproved observation model
for super-resolution under affine motion,” .IEEE Transactionson Image
This work was supported by National Natural Science Processing, Vol. 15, No. 11, 2006, pp.3325-3337.
Foundation of China under grant 60972001 and National Key