Document Image Reconstruction on Mobile Using Onboard Camera

Document Sample
Document Image Reconstruction on Mobile Using Onboard Camera Powered By Docstoc
					                           LYU0503
Document Image Reconstruction
          on Mobile
   Using Onboard Camera
Supervisor:
 Professor Michael R.Lyu

Group Members:
 Leung Man Kin, Stephen
 Ng Ying Kit, Tommy
              Outline
•   Motivation
•   Objective
•   Symbian Operating System
•   Previous Work
•   Implementation
•   Experimental results
•   Difficulties
•   Conclusion

                               2
               Motivation
• Most people posses mobile phones

• Quality of onboard camera improves
• Processing power increases

• Mobile phones have many functions
  – Take photos, Music Jukebox
  – How to add more values?
                                       3
              Objective
• Reconstruct documents on mobile phones
• Input: Different parts of document
• Output: A high resolution document




                                           4
     Symbian Operation System
•   A mobile OS from Symbian Ltd.
•   Design for 2G, 2.5G and 3G mobile phones
•   32-bit preemptive multi-tasking OS
•   Features:
    –   Multi-tasking kernel
    –   Mobile telephony support
    –   Communication
    –   Data management
    –   Etc.

                                               5
   Symbian Operating System
• Symbian phone models
  – Nokia 6600
  – Nokia 6630
  – Nokia N90




• Symbian SDK provided to develop program
• It enables programmers to gain more control on
  phone compared with java environment
                                                   6
   Symbian Operating System
• Some testing results using Nokia N90
  – It can allow use allocate at most 1M memory
    manually (e.g. malloc in C).
  – It can allow us to open 8-10 2Mega pixel gray scale
    image (using class CFbsBitmap).


• Why different?
  – The architecture of Symbian OS is client-server.
  – malloc in C is for client side.
  – The class is in the server side.
                                                          7
              Previous Work
• Study basic idea of different image alignment
  and stitching techniques.

• Implemented & tested on PC
  – Use SIFT to find matching points
  – Stitch the image using the matching points

• Tested some related APIs on Symbian OS
  – Processing image
  – Fax API
  – Etc.

                                                  8
            Implementation
• The implementation is divided into 5 parts
  – Take photos of the document
  – Lens correction of the photos
  – Mark matching points of the photos
  – Stitch the photos
  – Optimize the photo




                                               9
 Take photos of the document
1. Take the whole document
  – Align the distance between the camera and
    document to take photo of the whole
    document.




                                                10
  Take photos of the document
2. Take the individual part of the document
  – The camera is taken closer to the document
    to take the required image



                 1           2

                 3            4
                                              11
     Graphical User Interface
• To instruct the user to take photo of the
  individual part of the document
  – Overlaying is used.




                                              12
      Graphical User Interface
• Overlaying
  – Output.R = (Picture1.R + Picture2.R) / 2
  – Output.G = (Picture1.G + Picture2.G) / 2
  – Output.B = (Picture1.B + Picture2.B) / 2
  – R = Red
  – G = Green
  – B = Blue


• The process is simple and quick
  – Complexity is O(Width * Height)
  – Involved one addition and one shift for each color
    component for each pixel.
  – Can be used in Symbian phone real-time

                                                         13
  Take photos of the document
• After taking photo of the whole document, divide
  the whole document into several part like
                    1     2

                    3     4




• Use Overlay to display both the individual part
  and the current frame from the camera
                                                    14
  Take photos of the document
• Before the overlaying, the size of both image should be
  the same
   – Cannot instruct the user to take photo correctly.
   – Resizing is used.
   – It would take time if interpolation is needed.

• Careful use of image size
   – The image resolution 1600 x 1200 (whole document)
   – The resolution from the camera 800 x 600
       • If bigger (the same as the image), the camera would be lagging
   – The wanted resolution is about 400 x 300.
   – To reduce 1600 x 1200 to 400 x 300 and 800 x 600 to 400 x 300
       • Just sub-sample the whole document.
       • No interpolation is needed

                                                                          15
  Take photos of the document




• User is instructed to take photos correctly.
                                             16
   Take photos of the document
• Pseudo code
Take phone of whole document
Divide the whole document into four
For each sub document
   While (there is next frame from camera)
        Sub-sampled sub-document
        Sub-sampled of the current frame
        Overlay the above two image and display
        if (User take photo of sub-document)
                 Store the photo
                 break

                                                  17
            Lens correction
• Photos suffer from lens distortion
  – Barrel type
  – Pincushion type




                                       18
            Lens correction
• Image transformation
  – Transform the images using the equations




                                               19
Lens correction




                  20
   Mark matching points manually
• In order to reconstruct the photos well
  – Need to find match points between photo
  – Transform the individual photos according to
    the whole document

• The finding and matching feature points
  using SIFT need much time



                                                   21
 Mark matching points manually
• On PC (3.0Ghz), it need about:
   – finding feature points:
       • 4~5min each image
   – matching feature points:
       • 4~5 min each pair of image
   – TOTAL = 45 min

• The processing power of Symbian phone is much less
  than the PC.
   – Nokia N90: RISC 220Mhz processer

• Mark the points manually would be faster and more
  reasonable.

                                                       22
  Mark matching points manually
• Ask the user to mark the matching points




                                             23
   Mark matching points manually
• Like overlaying, resize is needed.
   – Only sub-sampling is needed.


• We use (x, y) coordinates to store for each points.

• In previous example, we would obtain 16 pairs of (x, y)
  coordinates.
   – Each individual image contributes 4 pairs
   – There are 4 individual images.


• The coordinates are stored and used to transform the
  individual photos in the later stage.
                                                            24
 Mark matching points manually
• Pseudo code

For each sub-document
      Display left half of the sub-document and
             corresponding part of whole document
      Marking two pairs of points
      Display right half of the sub-document and
             corresponding part of whole document
      Marking two pairs of points
      Store the coordinates into the file.

                                                    25
Mark matching points manually



                        Marker
  Marked




                                 26
Stitch the photos


1. Find transform
Parameter Set 1




                    2. Transform corresponding
                    coordinates using set 1
                                                 27
           Stitch the photos
3. Find transform parameter set 2
by sub-image2 and transformed coordinates




                       4. Using set 2 to transform
                       whole sub-image2
                                                     28
           Stitch the photos

5. Similarly for all other sub-images



                       6. Stick them together
           and high resolution image formed!


                                           29
            Image Blending
• Bilinear Interpolation
  – A weighted mean of nearby pixel values




                                             30
            Image Blending
• Averaging Mask
  – Mean of pixel values of 8 neighbors




                                          31
            Image Blending
• Threshold
  – Quantize the pixel values by a threshold value




                                                 32
Experimental Results




                       33
Experimental Results




                       34
Experimental Results




                       35
Experimental Results




                       36
Experimental Results




                       37
                  Difficulties
• Limitation of processing speed
  – Image processing involves much computation
  – Processing in phone would take much time than PC.


• Limitation of memory
  – PC have much more memory and is allowed to use
    virtual memory.
  – Image processing, like image stitching, involves much
    memory


                                                        38
               Conclusion
• We have studied different stitching
  algorithms

• We have implemented the stitching
  algorithm on both PC & Symbian phone

• Defects are still exist in the resulting image

                                              39
~ THE END ~


              40

				
DOCUMENT INFO