Estimation of Photo-Shoot Location with a Mobile ... - Aist-ari.org

Document Sample
Estimation of Photo-Shoot Location with a Mobile ... - Aist-ari.org Powered By Docstoc
					Estimation of Photo-Shoot Location
   with a Mobile-Phone Camera
    Based on Virtualized-Reality
       Environment Models

 Jun NISHIDA12, Tomoya ISHIKAWA1, Koji MAKITA1,
     Masakatsu KOUROGI1, Jun YAMASHITA2,
      Hideaki KUZUOKA2, Takeshi KURATA12

     1Center for Service Research, AIST, Japan
           2University of Tsukuba, Japan

                                     National Institute of Advanced Industrial Science and Technology
      EBS, Post POS, MR Information Sharing
• Real-virtual correspondences of products with POS
  (Point-Of-Sales) systems
    – Facilitate modeling and designing the flow of the products by
      not strongly relying on tacit knowledge.
    – Brought about drastic changes in retail, chain restaurant,
      logistics, etc.
    – Realized EBS (Evidence-Based Service) to some extend.

• On the analogy…
• One of the next key issues for service innovation
   How to Make better correspondence between
  customers/employees/service processes/environment
  and the computerized ones.
   Mixed-Reality Information Sharing technologies!
2                                               National Institute of Advanced Industrial Science and Technology
           MR Information Sharing Technologies in
    Human-Behavior Sensing and Visualization Research Team




3                                       National Institute of Advanced Industrial Science and Technology
     Horizontal-Vertical Development of
    MR Information Sharing Technologies




4                           National Institute of Advanced Industrial Science and Technology
        PDR(Pedestrian Dead-Reckoning)
    Estimates velocity vector, relative altitude, and actions
    by measurements from waist-mounted sensor module.
 Wearing sensor module on waist
   Easy to wear and maintain
   Easy to measure data for action recognition
   Relatively easily to apply for handheld
     setting compared to shoe-mounted PDR
     based on Zero Velocity Updates (ZUPTs)


        Sensor module
       ・Accelerometers
       ・Gyro-sensors
       ・Magnetometers
       ・Barometer                                 From PDR to PDRplus

                              Handheld PDR
5                                                 National Institute of Advanced Industrial Science and Technology
           Pedestrian Tracking System

                          Sensor module
                                       Accelerometers                 RFID                    Active                   Surveillance
    Barometer   Magnetometer           / gyro-sensors                reader           ID     RFID tag                    camera
           Atmospheric     Magnetic               Acceleration /        ID                                                          Video
           pressure        vector                 angular velocity      RSSI
                                                                           Walking
                                                                           velocity    Trajectory matching/
         Pedestrian dead-reckoning                                                      Velocity estimation
                            Velocity vector /
                            Relative altitude /                RF-based
                            Action type                        positioning
                                                                         Absolute
                                                                         position

                         Sensor/Data fusion                                            Trajectory
                                                                                                     3D model           3D environment
                           (Particle filter)                                                                                model
                                        Position /
                                        Orientation


                                Output of
                           position/orientation           T. Ishikawa, M. Kourogi, T. Okuma, and T. Kurata, “Economic and Synergistic
                                                          Pedestrian Tracking System for Indoor Environments”, In Proc. SoCPaR2009,
                                                          pp.522-527, 2009.
6                                                                                             National Institute of Advanced Industrial Science and Technology
                                   Our goal
Estimation of photo-shoot location with mobile-phone camera for
Initialization and Drift Cancellation.
                               Drift                                Drift
                                       Pass(estimated)
      Pass(estimated)
                    Cancellation                  Cancellation

                                                                              Pass (ground truth)
Initialization




7                                                         National Institute of Advanced Industrial Science and Technology
          Previous works~ Panorama-Based Annotation
                                                                            Direction
                                                                                   Position
                    Input frames
                                                                                             A


From the
                                                                                             B
user’s camera
                                                                                           5 [deg]
                                                                                            C
                      Located          Orientated          235 [deg]

                           G                         F
                                                               Position at which
                     B             C          D                a panorama is taken
                A
                                          E       Environmental map
                                                                              M. Kourogi, T. Kurata, et al.,
8                                                                             IWAR1999, ISWC2001



      ✔ A lot of panoramic images are needed for wide environments
                                                                National Institute of Advanced Industrial Science and Technology
          Panorama-Based to Model Based


                              Matching




         Input image                     Panoramic images




                       Matching



    Input image                                    Virtualized-Reality
9                                                 Environment Models
                                          National Institute of Advanced Industrial Science and Technology
        Design for coarse to fine localization
     Input Image                    Database
                                                                                                   Result
                              Photo-realistic virtualized-
                             reality environment models



         Coarse estimation                                          Fine estimation



                                         Coarse estimation result
           SSD                                                                   SIFT
                   ZNCC

          Histogram                                                             Edge
                   y
          similarity

10                                                                   National Institute of Advanced Industrial Science and Technology
                 Set up for experiments
     • Similarities for coarse estimation
       – Histogram similarity in the HSV space (using H and S)
       – Sum of Squared Difference (SSD)
       – Zero-mean Normalized Cross-Correlation (ZNCC)
     • Feature for fine estimation
       – Scale-Invariant Feature Transform (SIFT)



     • Images for experiments
       – Real images (photos taken with an iPhone)
       – Generated images from a virtualized-reality model

11                                           National Institute of Advanced Industrial Science and Technology
        Real images and generated images
                 for experiments
     Real images
                   A        B                                            C




     Generated images
                        A   B                                           C




12                              National Institute of Advanced Industrial Science and Technology
                      Image generation
                from a virtualized-reality model
                                    Position and orientation
                                               g g
                                   used for image generation




     Generated image from a
     virtualized-reality model




13                                        National Institute of Advanced Industrial Science and Technology
     Generated image from a virtualized-reality model
        (generated by changing camera position)




14                                   National Institute of Advanced Industrial Science and Technology
                              Similarity map
     Shooting position and orientation
             of a real image




Y




                   X
        Similarity map (ZNCC)            Real image taken with an Generated image from a
                                         i-Phone                  virtualized-reality model
15                                                           National Institute of Advanced Industrial Science and Technology
                       Experimental result :
       Histogram similarity in the HSV space (using H and S)
                        A                                 B                                             C
       Real images




                                          Similarity of the nearest point is NOT high.
     Generated images




16                                                                             real image
                            : Shooting position and orientation of a of Advanced Industrial Science and Technology
                                                            National Institute
                        Experimental result :
          Similarity of Sum of Squared Difference (SSD)
       Real images      A                            B                                             C




                                                 Similarity of the nearest point is low.
     Generated images




17                                                                             real image
                            : Shooting position and orientation of a of Advanced Industrial Science and Technology
                                                            National Institute
             Experimental result : Similarity of
      Zero-mean Normalized Cross-Correlation (ZNCC)
       Real images      A                                 B                                                 C




                                                        ・Similarity of the nearest point is
     Generated images                                   comparatively high.




18                                                                             real image
                            : Shooting position and orientation of a of Advanced Industrial Science and Technology
                                                            National Institute
     Generated image from a virtualized-reality model
        (generated by changing camera position)




19                                   National Institute of Advanced Industrial Science and Technology
                              Experimental result :
              Histogram similarity in the HSV space (using H and S)

             0.25               図5,Color      図6,Color      図7,Color
                                                                C
                                   A             B
Similarity




              0.2

             0.15

              0.1

             0.05                                                 Similarity of the nearest
                                                                     point is NOT high.
                0

             -0.05
                  -100            -50              0              50                                   100
                         Difference of shooting orientations between real image and
                         generated image (degree)
20                                                                 National Institute of Advanced Industrial Science and Technology
                              Experimental result :
                Similarity of Sum of Squared Difference (SSD)
                         図5,SSD
                            A            図6,SSD
                                            B           図7,SSD
                                                           C
              0.9
Similarity




             0.85
              0.8
             0.75
                                                     Similarity of the nearest
              0.7                                       point is NOT high.
             0.65
              0.6
             0.55
              0.5
                 -100        -50             0              50                                100
                        Difference of shooting orientations between
                        real image and generated image (degree)
21                                                               National Institute of Advanced Industrial Science and Technology
                    Experimental result : Similarity of
             Zero-mean Normalized Cross-Correlation (ZNCC)
                       図5,ZNCC
                          A          図6,ZNCC
                                        B           図7,ZNCC
                                                        C
Similarity




             0.5
             0.4
             0.3                                 Similarity of the nearest point
                                                 is comparatively high.
             0.2
             0.1
              0
         -0.1
         -0.2
         -0.3-100       -50              0               50                                100

                       Difference of shooting orientations between
                       real image and generated image (degree)
22                                                         National Institute of Advanced Industrial Science and Technology
     Sample result of SIFT matching
        Generated image             Generated image
     (Almost same position)        (Different position)




       Real image             Real image
23                                         National Institute of Advanced Industrial Science and Technology
       Conclusion of coarse estimation

     • HSV : NOT Appropriate
       – Similarity of the nearest point is NOT high.
     • SSD : NOT Appropriate
       – Similarity of the nearest point is NOT high.
     • ZNCC : Appropriate
       – Similarity of the nearest point is local maximal.




24                                                 National Institute of Advanced Industrial Science and Technology
                    Future works
• More Experiments
     – in wide environments
     – with changing position and orientation in same time
• Observational study for fine estimation
     – More study with SIFT
     – Study with edge based method




25                                        National Institute of Advanced Industrial Science and Technology
                   Appendix 1 :
             Definitions of similarities


                  ∗       ∗

                  ∑           ∑       ,                    ,

              ∑   ∑               ,       ∗∑   ∑                       ,


     ∑   ∑            ,           ∑   ∑            ,

     ∑   ∑            ,           ∑   ∑                ,


26                                             National Institute of Advanced Industrial Science and Technology
                      Appendix 2 :
 Similarity maps of ZNCC before and after the earthquake
     A




                         Before




27                        After
                                       National Institute of Advanced Industrial Science and Technology
                      Appendix 2 :
 Similarity maps of ZNCC before and after the earthquake
     B




                         Before




28                        After
                                       National Institute of Advanced Industrial Science and Technology
                      Appendix 2 :
 Similarity maps of ZNCC before and after the earthquake
     C




                         Before




29                        After
                                       National Institute of Advanced Industrial Science and Technology

				
DOCUMENT INFO
Categories:
Tags:
Stats:
views:0
posted:4/7/2013
language:Unknown
pages:29