# EE 7730 Lecture 1

Document Sample

```					EE 7700

Demosaicking Problem in Digital Cameras
Multi-Chip Digital Camera
     To produce a color image, at least three spectral
components are needed at each pixel.
     One approach is to use beam-splitters and multiple chips.

Lens

Scene

Beam-
splitters   Spectral   Sensors
filters

Single-Chip Digital Camera
     Multi-chip approach is expensive. Precise chip alignment is
required.
     The alternative is to use a color filter array.

Lens
Color filter   Sensors
array

Scene

Single-Chip Digital Camera
     The missing color samples must be estimated to produce
the full color image.
     Since a mosaic of samples are available, this estimation
(interpolation) process is called demosaicking.

Single-Chip Digital Camera
     Images suffer from color artifacts when the samples are not
estimated correctly.

Original image        Bilinearly interpolated
from CFA-filtered samples
Demosaicking Approaches
each color channel separately using a standard technique,
such as nearest-neighbor interpolation, bilinear
interpolation, etc.
     Edge-Directed Interpolation: Estimate potential edges,
avoid interpolating across the edges.

Edge-directed interpolation

1.Calculate horizontal gradient ΔH = |G1 – G2|
3       2.Calculate vertical gradient ΔV = |G3 – G4|
1   x   2   3.If ΔH > ΔV,
Gx = (G3 + G4)/2
4
Else if ΔH < ΔV,
Gx = (G1 + G2)/2
Else
Gx = (G1 + G2 + G3 + G4)/4

Demosaicking Approaches
     Edge-Directed Interpolation: Based on the assumption that
color channels have similar texture, various edge detectors
can be used.

1           Edge-directed interpolation
2
1.   Calculate horizontal gradient ΔH = | (R3 + R7)/2 – R5 |
3   4    5   6   7
2.   Calculate vertical gradient ΔV = | (R1 + R9)/2 – R5 |
8           3.   If ΔH > ΔV,
9                   G5 = (G2 + G8)/2
Else if ΔH < ΔV,
G5 = (G4 + G6)/2
Else
G5 = (G2 + G8 + G4 + G6)/4

Demosaicking Approaches
     Constant-Hue-Based Interpolation: Hue does not change
abruptly within a small neighborhood.
   Interpolate green channel first.
   Interpolate hue (defined as either color differences or color
ratios).
   Estimate the missing (red/blue) from the interpolated hue.

Red                              Interpolate            Interpolate
d Red

Green       Interpolate

Demosaicking Approaches
     Edge-Directed Interpolation of Hue: It is a combination of
edge-directed interpolation and constant-hue-based
interpolation. Hue is interpolated as in constant-hue-based
interpolation approach, but this time, hue is interpolated
based on the edge directions (as in the edge-directed
interpolation algorithm).

Demosaicking Approaches
     Using Laplacian For Enhancement: Use the second-order
gradients of red/blue channels to enhance green channel.

1.   Calculate horizontal gradient ΔH = |G4 – G6| + |R5 – R3 + R5 – R7|
1            2.   Calculate vertical gradient ΔV = |G2 – G8| + |R5 – R1 + R5 – R9|
2            3.   If ΔH > ΔV,
G5 = (G2 + G8)/2 + (R5 – R1 + R5 – R9)/4
3   4   5    6   7
Else if ΔH < ΔV,
8
G5 = (G4 + G6)/2 + (R5 – R3 + R5 – R7)/4
9                 Else
G5 = (G2 + G8 + G4 + G6)/4 + (R5 – R1 + R5 – R9 + R5 – R3 + R5 – R7)/8

Aliasing                                     f2

Frequency spectrum of an image:
fm     f1

After CFA sampling:
f2
f2

f1                                f1

Green channel                     Red/Blue channel

Demosaicking Approach
     Alias Cancelling: Based on the assumption that red, green,
and blue channels have similar frequency components, the
high-frequency components of red and blue channels are
replaced by the high-frequency components of green
channel.

f2

f1

Red/Blue channel

Experiment

HL    HL    HL
Full                         LL
Subband            LL    LL
Red/Green/Blue
channels
decomposition                    HH
LH    LH    LH

CFA
Sampling

Interpolate                        HL    HL    HL
Subband      LL    LL    LL
decomposition                    HH
LH    LH    LH

Constraint Sets

 Detail Constraint Set: Detail subbands of the red and blue
channels must be similar to the detail subbands of the green
channel.

GHL
HL    HL            RHL
HH    HH
LH    LH

 R(n1 , n2 ) : Rk (n1 , n2 )  Gk (n1 , n2 )  T (n1 , n2 ),
                                                            
Cd                                                              
 for k  HL, LH , HH
                                                            


Constraint Sets

 Observation Constraint Set: Interpolated channels must be
consistent with the observed data.
Sensors    CFA

O(n1 , n2 )
R

Co  R(n1, n2 ) : R(n1, n2 )  O(n1, n2 ),  (n1, n2 ) R 

Projection Operations

 Projection onto the Detail Constraint Set:
 Decompose the color channels.
 Update the detail subbands of red and blue channels.

GHL (n1 , n2 )

HL

HH                         RHL (n1 , n2 )
LH
GHL (n1 , n2 )  T (n1 , n2 )

 Apply synthesis filters to reconstruct back the channels.

Projection Operations

 Projection onto the Observation Constraint Set:
 Insert the observed data to their corresponding positions.

Sensors    CFA

O(n1 , n2 )

Alternating Projections Algorithm

Samples of                         Initial
color channels                  interpolation

Projection onto the              Projection onto the
detail constraint set         observation constraint set

h0                      g0
Insert the
observed data
h1        Update        g1

Iteration

Results

Original      Hibbard 1995   Laroche and Prescott 1994

Hamilton and Adams 1997   Kimmel 1999         Gunturk 2002

Results

Laroche
and
Hibbard   Prescott
Original                 1995      1994

Hamilton
and
1997                   1999      2002
Previous Methods

[Gunturk02]

Gunturk et al, “Demosaicking: Color Filter Array Interpolation in Single-Chip Digital
Cameras,” to appear in IEEE Signal Processing Magazine.

References
 [Gunturk02] Gunturk et al, “Color Plane Interpolation Using Alternating
Projections,” IEEE Trans. Image Processing, 2002.
[Hibbard 1995] R. H. Hibbard, “Apparatus and method for
adaptively interpolating a full color image utilizing luminance
gradients,” U.S. Patent 5,382,976, January, 1995.
 [Laroche and Prescott 1994] C. A. Laroche and M. A. Prescott,
“Apparatus and method for adaptively interpolating a full color
image utilizing chrominance gradients,” U.S. Patent 5,373,322,
December, 1994.
 [Hamilton and Adams 1997] J. F. Hamilton Jr. and J. E. Adams,
“Adaptive color plane interpolation in single sensor color electronic
camera,” U.S. Patent 5,629,734, May, 1997.
 [Kimmel 1999] R. Kimmel, “Demosaicing: Image reconstruction
from CCD samples,” IEEE Trans. Image Processing, vol. 8, pp.
1221-1228, 1999.