Docstoc

Image Processing Digital Cameras

Document Sample
Image Processing Digital Cameras Powered By Docstoc
					IEEE SCV SPS                            r.kakarala@ieee.org




               Image Processing
                        for
                Digital Cameras

                 Ramakrishna Kakarala

                 Agilent Technologies


                         1
IEEE SCV SPS                                           r.kakarala@ieee.org




                           Outline

        1. The digital still camera (DSC) market
        2. Anatomy of DSC - a signal processing view
        3. Image pipeline in a DSC
              o cleaning up the acquisition
              o color interpolation
              o color correction
              o display compensation
              o compression
        4. Frontier areas - research topics
        5. Conclusions


                                2
IEEE SCV SPS                                                         r.kakarala@ieee.org




                 Digital still cameras

                                                              www.dpreview.com

        Typical features ($300-$1,000)
           o 3-5 Million pixels (megapixels), i.e, 2588x1954
           o Flash memory storage (CF, SD, MS)
           o Video clip capture (MPEG, Quicktime)
           o LCD preview

        Large and growing market
           o 14M units (2003)
           o digital > film (unit vol) in 2008
                           www.infotrends-rgi.com/Press/itPress/2003/6.25.03
                                   3
IEEE SCV SPS                                  r.kakarala@ieee.org




                 Embedded DSCs

               Cellphone cameras appeal to
                  o phone makers
                  o wireless operators
                  o most new designs

               What are their uses?
                 o journalism
                 o “always-with-you” camera
                 o security

               Webcams: a lesson
                             4
IEEE SCV SPS                          r.kakarala@ieee.org




       “Anatomy” of digital still camera
                                      LCD



                                     Digital
                               A/D   Image
                                     Pipeline


                      sensor

                                     Storage
               lens


                       5
IEEE SCV SPS                                            r.kakarala@ieee.org




                          Sensor layout




                                   Color
                                   filter array
               filters

       Three sensors:               One sensor:
       Expensive, bulky             Requires interpolation




                               6
   IEEE SCV SPS                                                      r.kakarala@ieee.org




                     Sensor layout variations
              microlenses                                         3 color pixels




micron.com/products/imaging/technology/pixel.html

                                                             www.x3f.info
                                    Hexagonal
                                    mosaic
                                                                             CMY


                                                        www.dpreview.com
          home.fujifilm.com
                                                    7
IEEE SCV SPS                                      r.kakarala@ieee.org




               Digital camera image pipeline

  Processing stages after A/D
        a) Photo-response non-uniformity (PRNU)
        b) White balance
        c) Color interpolation
        d) Enhancements: sharpen, noise-clean
        e) Color correction
        f) Tone mapping

                                 8
IEEE SCV SPS                                            r.kakarala@ieee.org




                     Response correction
  ●
      Illumination variation : cos^4
        –      off-axis illumination reduced by cos(q)^4
        –      removable by calibration, spatial gain
  ●
      Bad pixel correction
        –      “Stuck-on”, “stuck-off” pixels
        –      “hot” pixels visible in low-light/long exposure


                                    9
IEEE SCV SPS                                        r.kakarala@ieee.org




               Bad pixel removal
  ●
      Simple: thresholding, median filtering
  ●
      Sequential analysis   (Tan   & Acharya,ICASSP '99 )

                       Algorithm
                          a) Calculate in neighboorhood
                                 S =min∣X − Xn∣

                            b) if S > T consistently ==>
                             bad pixel


                            10
 IEEE SCV SPS                                 r.kakarala@ieee.org




                       Bad pixel correction




www.mediachance.com/dce/




                                11
IEEE SCV SPS                                         r.kakarala@ieee.org




               Color imaging

                 I ( )

                               sk   I ( ) R( ) wk (  ) d,
                               k  1,2,3

R( )

                                       w k 




                               400                     700
                          12
    IEEE SCV SPS                                                                         r.kakarala@ieee.org




                    Color temperature

              3


                                    Incandescent : T=2500K
             2.5




                                                                                         [                      ]
                                                                                                                −1
Normalized    2                                                                 c1                  c2
spectral                                                             I B =           ∗ exp            −1
power                                                                           
                                                                                    5
                                                                                                  ∗T
             1.5




              1



                                           Daylight: T=6500K
             0.5




              0
              400    450   500       550      600    650       700

                                 Wavelength (nm)
                                               13
IEEE SCV SPS                                                      r.kakarala@ieee.org



                          White balance
  Camera set for incandescent                                      flourescent




                                     Nikon Coolpix 990: Source Prof. C K Shene
                                14
IEEE SCV SPS                                                    r.kakarala@ieee.org




                          White balancing


         Choose gains c1, c2, c3 to compensate for illuminant



                              c1

                                           c2
                  s1          x
                                                      c3
                               s2         x

                                          s3          x

                                     15
IEEE SCV SPS                                                             r.kakarala@ieee.org




                         Auto white balance

   Many algorithms:
         –     Gray world: mean R, G, B values should be equal
         –     Correlation method (Tominaga-Wandell)
                   Convex hull of R,B values matched to gamut of illuminant
                 color temperature.


   Real-world problems


         –     Large uniform patches of color
         –     Multiple illuminants
         –     Non standard bulbs


                                            16
IEEE SCV SPS                                                             r.kakarala@ieee.org




                             Demosaicking
   • Color from monochromatic sensors: two methods




                                     Color filter
                                     array
                   filters

          Three sensors:                   One sensor:               “Demosaiced” color
          Expensive, bulky                 Requires interpolation    image



                                 Research topics:
                                    (a) interpolation algorithms
                                    (b)color filter array patterns



                                      17
IEEE SCV SPS                                                           r.kakarala@ieee.org


                          Demosaicking
                             What the sensor sees: mosaiced      Demosaiced image: note
               Original
                             image                               color aliasing




                                                              Bilinear interpolation

                                        18
IEEE SCV SPS                                                                r.kakarala@ieee.org




               Demosaicking algorithms



                                                                                 Method:
                                                                                 G=B-Bav+Gav




                       Simple (bilinear or nearest   Estimate high frequencies
                       neighbour) interpolation      and incorporate




                                   19
IEEE SCV SPS                                   r.kakarala@ieee.org




    original   PVM (Kakarala-Baharav)   Hamilton-Adams (kodak)




                          20
IEEE SCV SPS                             r.kakarala@ieee.org




               Idea: detect edges in color
      ●
           Edges: locations of
           sharp changes--they
           occur in luminance &
           chrominance
                                        Luminance edge




      ●
           Always interpolate
           along edges, rather
           than across them             Chrominance edge
                                        (isoluminant)




                                  21
IEEE SCV SPS                                                      r.kakarala@ieee.org




                             Detecting color edges

       In monochrome
                                                             
   ●
                                                             y

       images, use gradient
                                                        G
                     I 
                     
               I   x 
                      I      I  T  edge                       R
                     
                     y 
                     
                                                   B                          
   ●
       In color images, 3                                                      x


       gradients

                R  G B




                                              22
 IEEE SCV SPS                                                    r.kakarala@ieee.org




                Interpreting color gradients

    ●
        Form Jacobian
                     R    G    B 
                         x    x   x 
                                  B   R  G  B
                 J  R
                      y   G
                              y   y 


                                                            u2
    ●
        SVD of Jacobian                                           u1


                          t              t              t
J  USV t   1u1v1   2u2 v2   1u1v1
                                                                     u1  u2




    ●   U1 = principal vector; U2 = residual vector
                                                 23
IEEE SCV SPS                                                     r.kakarala@ieee.org




                Principal & residual vectors

   ●
       Principal vector -> steepest change;
       residual vector -> least change
                                                       u2   u1


   ●
       Interpolate along both vectors + add
                                                                 I
                    1               2
               I          I res           I prin
                  1   2         1   2
                                                     Ires                Iprin




                                            24
IEEE SCV SPS                                      r.kakarala@ieee.org




 •       Interpolation along residual, principal
              Implementation issues
         vectors not usually possible
               -- replace with horizontal, vertical
                 interpolation


 •       Scheme requires SVD
               -- develop simple approximations




                                 25
IEEE SCV SPS                                                                r.kakarala@ieee.org




                         Simple interpolation

   ●
       Add horizontal & vertical interpolants
       I  I res  (1   ) I prin              I horiz   
                                                             I 2  I 4 
                                                                   2
       
                                                 I vert    
                                                              I1  I 3 
       I  I horiz  (1   ) I vert                             2
                    I1                  Choose  based on principal vector
               I4        I2


                    I3




                                            26
IEEE SCV SPS                                          r.kakarala@ieee.org




               Simple interpolation (cont’d)

   ●
       Interpolation rule
                     I  I horiz  (1   ) I vert

                                             1
   ●
       Quantize so that                   0,1, 
                                             2




                                      27
IEEE SCV SPS                                                 r.kakarala@ieee.org




               Choosing quantized weight
Form Jacobian

                   R    G    B 
               J   x      x    x   R  G  B
                   R y G y B y 
                                     

Count # of times                | y || x |
Assignment: if (0) or (1)                   0    (vertical)
                                                                        “Majority
                     if (2) or (3)           1   (horiz)
                                                                        Rule”




                                28
IEEE SCV SPS                                             r.kakarala@ieee.org




                    Additional case

   ●
       Add components of Jacobian

                    J  R  G  B



                         1
               J T       (average horiz. and vertical)
   ●
       If                2
       else use previous rule to set                 0,1



                               29
IEEE SCV SPS                                              r.kakarala@ieee.org




                   Why majority rule?
    ●
         Good predictor of principal vector
                                  t          t        t
               J  USV t   1u1v1   2u2 v2   1u1v1


        majority |dy|>|dx| --> |u1(2)|>|u1(1)|?


        accurate 91 % of time (1000 trials)


                                      30
IEEE SCV SPS                                                        r.kakarala@ieee.org




               Why majority rule? (cont’d)
   ●
       Allows mix of 1st and 2nd derivatives
                    n R        G     m B 
                ~        x n      x    x  ; n, m  1, 2
                                                m
                J  n
                    R          G     B m
                                        m

                   
                         y n      y    y 
                                             

   ●
       majority |dy^n|>|dx^n|
   ●
       both 1st and 2nd derivs can be estimated
       in some locations, e.g.,
                                                       I0      I1     I2
       dx = I2-I0; dx^2 = 2I1-I2-I0


                                           31
IEEE SCV SPS                            r.kakarala@ieee.org



               PVM        Hamilton-Adams (kodak)




                     32
IEEE SCV SPS                                                      r.kakarala@ieee.org




                          Noise cleaning
       Many algorithms:
                                                             X1   X2   X3
        1) Sigma filtering (J. S. Lee, 1982)                 X4   Xc   X6
                                                             X7   X8   X9
               X = mean{Xi: |Xi-Xc| < }

        2) Global and local  estimates (Kodak patents)

        3) Vector median of a set: smallest total distance

                                   {        9
                                                         }
                     VM =argmin w X w : ∑k =1 ∥ X k − X w∥




                                       33
IEEE SCV SPS                                 r.kakarala@ieee.org




               After market noise cleaners




                           34
IEEE SCV SPS                                          r.kakarala@ieee.org




               www.michaelalmond.com/Articles/noise



                                          35
IEEE SCV SPS                                          r.kakarala@ieee.org




               www.michaelalmond.com/Articles/noise



                                          36
IEEE SCV SPS                                                                         r.kakarala@ieee.org




                       Color correction

                               CIE XYZ                                            sRGB
     Sensor response        (device independent)                                 (display)




                         http://www.creativepro.com/story/feature/13036-2.html




                                           37
IEEE SCV SPS                                      r.kakarala@ieee.org




       Color correction: Sensor to XYZ

        “Maximum ignorance” method

                 Solve: W =ST  T = pinv S W

                 W = n x 3 XYZ responses
                 S = n x 3 sensor RGB responses
                 T = 3x3 transformation matrix

        Surface based (Vrhel-Trussell)

        Bayesian (Brainard-Freeman)

        White-point preserving (Finlayson-Drew)

                                      38
IEEE SCV SPS                                                                         r.kakarala@ieee.org




               A comparison of 3 consumer digital cameras




                             http://www.imaging-resource.com/PRODS/D30/3TPICS1.HTM




                                        39
IEEE SCV SPS                             r.kakarala@ieee.org




        A Digital Camera Image Pipeline
               AWB           Demosaic          CC
  sensor


                                        Tone
                     Compression
 exposure                               mapping
 setting



                       Storage


                        40
IEEE SCV SPS                                                      r.kakarala@ieee.org




                           Tone mapping
   Gamma correction


      Display response                Compensation

               y  x                       x  s1 / 

               Typical value =2.2


   Tone mapping: best use of display dynamic range


         Ex: Nikon D1 has 12 bits/pixel ---> Display is 8 bits.

                                       41
IEEE SCV SPS                                                                      r.kakarala@ieee.org




               Tone mapping methods

      Simple: Histogram equalization

      Others: Adaptive logarithmic (Drago, et al 2003)

                        display      Ld  L w world

                          0.01∗Ldmax               ln  L w 1
                 Ld =
                        log 10  L wmax 1               Lw
                                                                       ln b
                                                                      ln 0.5 
                                              ln 28                           
                                                         L wmax



                                      42
IEEE SCV SPS                     r.kakarala@ieee.org




               before tone map

                     43
IEEE SCV SPS                    r.kakarala@ieee.org




               after tone map

                       44
IEEE SCV SPS                                  r.kakarala@ieee.org




    Fusion of multiple exposure images (Drago et al)




                           45
IEEE SCV SPS                                             r.kakarala@ieee.org




                      “Frontier” Areas

   More stages in pipeline
         Image fusion (focus and exposure)
         Optical distortion removal (narrower cameras)


   Host vs embedded implementation

   A standard pipeline?



                                     46

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:9/2/2011
language:English
pages:46