Epipolar Image Generation and Corresponding Point Matching from by gdf57j

VIEWS: 12 PAGES: 6

									    EPIPOLAR IMAGE GENERATION AND CORRESPONDING POINT MATCHING
                 FROM COAXIAL VEHICLE-BASED IMAGES
                                                 Zhizhong Kang
                                         Faculty of Aerospace Engineering
                                          Delft University of Technology
                                   Kluyverweg 1, 2629 HS Delft, The Netherlands
                                                Z.Kang@tudelft.nl


ABSTRACT
The matching of corresponding points is the foundation of image-based 3D reconstruction. As coaxial vehicle-based
images as concerned, however, since they are taken along the optical axis epipolar lines of coaxial stereo pair are
arranged in the direction of radial in stead of nearly horizontal arrangement of stereo pair with horizontal baseline.
Therefore, the generation of epipolar images and corresponding point matching on epipolar images are discussed in
this paper. Although the straight lines in raw image become arcs in epipolar image, the point features are still
apparently clear and possible for 1D corresponding point matching. Moreover, matching in the epipolar images can
improve the success rate. Coaxial vehicle-based images are used to test the algorithms presented in this paper.


                                               INTRODUCTION

                                                   Image sequences taken along the optical axis, presently, are
     Z      Y                                 involved indispensably in localization of Mars rovers, mobile mapping,
                                              site modeling, surveillance applications and especially moving robot
     S1           X                           navigation. Therefore, it becomes a hot issue in photogrammetry and
                                              computer vision communities to research how to process this kind of
             a1                               image sequence. There are two ways to form stereo vision along the
                                              optical axis in general: one is to take images by a single camera moving
     BZ
           S2                                 along the optical axis (popular in localization of Mars (Ron Li et al.,
         BY
                                              2002) rovers and mobile mapping); the other is to align coaxially two
        BX
                                              omnidirectional (Gluckman et al., 1998 & Lin et al., 2003) (generally in
                      a2                      moving robot navigation). The research presented in this paper is carried
                                  A           out on the stereo vision formed in the former way, i.e. a single camera
    Figure 1. components of baseline.         moving along the optical axis.
                                                   Generally, as aerial photogrammetry as be concerned, the main
component, max(BX, BY, BZ), of horizontal baseline is commonly BX (Wang, 1990). Therefore the geometry of stereo
pair with horizontal baseline is naturally evolved under this hypothesis. For the stereo pairs along the optical axis,
however, the components of the baseline B should satisfy such condition: BZ>>BX, BZ>>BY in strictly speaking
(Fig.1.). In this case, the geometry of stereo pair will be distinctly different compared with that of horizontal
baseline. As we know, photogrammetric computer vision is concerned the reconstruction of objects via the stereo
pairs structured by multiple view photography. In principle, the corresponding point matching in the image is 2D,
however, the 2D matching can be convert to 1D by introducing corresponding epipolar lines (Zhang et al., 1997). As
aerial photogrammetry as be concerned, the arrangement of epipolar lines is nearly horizontal because the stereo pair
has horizontal baseline. In coaxial stereo pairs, however, epipolar lines in images along the optical axis are arranged
in the direction of radial because epipoles lie in the image near the principal point. As a result, we cannot expect to
implement 1D correlation in the raw image.
     Therefore, according to the epipolar geometry of coaxial stereo pair, we will discuss the generation of epipolar
image and corresponding point matching on it in this paper.




                                         ASPRS 2008 Annual Conference
                                   Portland, Oregon April 28 - May 2, 2008
                                   EPIPOLAR IMAGE GENERATION

                                                                  As aerial photogrammetry as be concerned, the
                        S1                                   arrangement of epipolar lines is nearly horizontal because
                                                             the stereo pair has horizontal baseline. In coaxial stereo
                   I1                                        pairs, however, epipole is the intersection of baseline S1S2
                                                             and image. In this case, epipoles lie in the image near the
                                                             principal point as illustrated in Fig. 2. As we know,
                                       Y                     epipolar lines are lines passing both epipoles and image
                  S2
                                                             points, e.g. I1 a1, I2 a2. The epipolar lines in images along
                  Z
                                                             the optical axis are arranged in the direction of radial
                                                             because epipoles lie in the image near the principal point.
                  I2                                              Relative orientation is implemented on coaxial stereo
                                                             pairs and the orientation elements computed are used to
                   I                                         determine the corresponding epipolar lines in stereo
                                                   X         image pair.
        Plane Z

                                                             Estimation of the Epipolar Geometry
  Figure 2. Epipolar geometry of coaxial stereo pair.             As mentioned in previous section, the geometry of
                                                             stereo pair is distinctly different with that of horizontal
baseline. In this case, relative orientation will be accordingly different with conventional photogrammetry as well.
Therefore, the formula of relative orientation of coaxial stereo pairs (Zhang and Kang, 2004) is employed to compute
the relative orientation elements. As we know, there are five elements of relative orientation and these elements must
be independent to each other. In coaxial stereo pairs, angle κ 1 and κ 2 are highly correlative instead of angle ω 1
and ω 2 as in traditional two-projector method of relative orientation, therefore the orientation elements should be
ϕ1, ω1, ϕ 2, ω 2, κ 2 .
     When relative orientation process is finished, the corresponding epipolar orientation vectors of an arbitrary
point in the left image can be computed using the relative spatial orientation of coaxial stereo pair. The equations of
corresponding epipolar lines in the stereo pair can be expressed as below.


                                                       Ax + By − Cf = 0
                                                    A' x '+ B ' y '−C ' f = 0                                      (1)


Where, ( A B C ) , ( A' B ' C ') are orientation vectors of corresponding epipolar lines in the left and right images
respectively. (x, y) and (x’, y’) are image coordinates in the left and right images respectively. f is the focus.
    Epipolar orientation vectors in the left and right images are computed by the formula below.


                                                                ⎡ x0 ⎤
                                                       vT
                                                       l = R BR ⎢ y 0 ⎥
                                                            T
                                                                ⎢ ⎥
                                                                ⎢− f ⎥
                                                                ⎣ ⎦
                                                                                                                   (2)
                                                                 ⎡ x0 ⎤
                                                     vT
                                                     l ′ = R' BR ⎢ y 0 ⎥
                                                             T
                                                                 ⎢ ⎥
                                                                 ⎢− f ⎥
                                                                 ⎣ ⎦

       v               v
Where, l = ( A B C ) , l ′ = ( A' B' C ') . R and R ' , rotation matrices of left and right image respectively,




                                         ASPRS 2008 Annual Conference
                                   Portland, Oregon April 28 - May 2, 2008
                                                                    ⎡ 0            − BZ     BY ⎤
                                                                    ⎢                      − B X ⎥ , BX, BY and BZ are
computed by relative orientation elements ϕ1, ω1, ϕ2, ω2 and κ2. B= ⎢ B Z            0           ⎥
                                                                    ⎢− BY
                                                                    ⎣               BX       0 ⎥ ⎦
baseline components.

Determination of the Area to Generate Epipolar Image
     In aerial photogrammetry, the area in the raw image to generate epipolar image is close to a rectangle because
the stereo pair has nearly horizontal epipolar lines. However, since the epipolar lines in coaxial stereo pair are
arranged in the direction of radial, the area to generate epipolar image becomes a sector and the center is the epipole.
Obviously, the closer to the epipole, the larger distortion in the epipolar image. As a result, a sector ring is chosen
instead of a simple sector. It is determined by the coordinates of a starting point and an ending point selected from
the raw image to generate epipolar image. The inner radius of sector ring should be determined to reduce the
distortion of epipolar image. Since the images are taken while the vehicle is moving forward, the scene included by
latter image is smaller than that included by the former. The area to generate epipolar image, accordingly, is
determined from latter image so that the corresponding areas are visible in both of the stereo image pair. The area in
former image can be determined using the orientation elements computed.

Generating Epipolar Image
     According to the area determined, the epipolar lines from the starting point to the ending point in raw image are
rearranged in the sequence of bottom-to-up with certain interval (e.g. 10') into epipolar image (Fig.3). In Fig.3,
although the straight lines in raw image become arcs in epipolar image, the point features are still apparently clear.
Therefore, the key point is that corresponding point matching is possible although the epipolar image differs
obviously from the raw one.




                                              Figure 3. Epipolar image.


                                CORRESPONDING POINT MATCHING
    The image matching algorithm, which is employed to find corresponding points between adjacent images, is
explained bellow.
    The first step is to build an image pyramid. Image pyramid (e.g. Kraus et al, 1997) is usually used for the
representation of a digital image in different resolution levels. Fig. 4 shows an example of an image pyramid. The
idea of image pyramids as a stack of digital images actually depicting the same scene with decreasing resolution is
closely related to the concept of scale space (Yuille and Poggio, 1986), in which the scale is introduced as an
additional (continuous) dimension of a digital image.




                                         ASPRS 2008 Annual Conference
                                   Portland, Oregon April 28 - May 2, 2008
                                                                                            Coarse




                                                                                               Fine



                                               Figure 4. Image pyramid.

     Image pyramids combine the advantages of both high and low resolutions of digital images. The lower levels of
an image pyramid provide detailed information, but a great amount of data, whereas the higher levels contain less
information but give an overview and require a smaller amount of data. The strategy of coarse-to-fine matching on
image pyramid is used to increase the convergence radius and thus avoid searching for correspondences in the row
images. Conversion to an image pyramid proceeds by recursively halving the resolution of images. As shown in
Fig.4, at each stage of the recursion, every 2 × 2 pixels is replaced by a single pixel that contains the average value
of those four pixels.
     After image pyramid is created, feature points are extracted. Generally, feature points can be extracted using
corner detectors such as Moravec operator (Moravec, 1977) or Harris/Pressley operator (Harris and Stephens, 1988).
The following text briefly outlines Moravec operator since it is implemented in our approach. Moravec has defined
the concept of ‘points of interest’ as being distinct regions in images and concluded these interest points could be
used to find matching regions in consecutive image frames. Moravec has proposed measuring the intensity variation
by shifting a small square window (typically, 3 × 3, 5 × 5, or 7 × 7 pixels) by one pixel in each of the eight principle
directions (horizontally, vertically, and four diagonals) on an image. The intensity variation for a given shift is
calculated by taking the average sum of squares of intensity differences of corresponding pixels in eight principle
directions (Moravec, 1977). The intensity difference, used as a threshold, determines feature points, i.e. those which
intensity difference is larger than the threshold. The principle of threshold selection is to keep real feature point and
exclude non-feature one. Since feature points have small neighborhoods, raw images are divided into grid. The size
of grid cell is selected with respect to the resolution and larger than the size of searching window. Intensity
difference for a given shift is computed by Eq. (3) (for a window of 3 × 3)

                                                              i +1    j +1

                                                             ∑∑
                                                          1
                                            M (i, j ) =                     ( I (k , l ) − I (i, j )) 2              (3)
                                                          8 k =i −1 l = j −1

where I(i,j) denotes the intensity of a given pixel (i,j).
     All potential feature points are picked out and the point with maximum difference is classified as a real feature
point. Thus, only one feature point is extracted in each grid cell to ensure reasonable distribution of feature points.
After extracting feature points in left image, image matching process is employed to find corresponding points in the
right one. Image correlation is used to find the point with maximum correlation coefficient as corresponding one.
     Image correlation (e.g. Kraus et al., 1997) is a technique by which the conjugate point of a slave image (right)
corresponding to the master image (left) is searched for the maximum correlation coefficient. The size of the
window should be selected depending on the image resolution and feature size. 9 × 9 to 21 × 21 would be better
used for digitized aerial photographs or close-range imagery.
     Smooth constraint is employed to detect and remove falsely accepted corresponding points. For every
corresponding point pair, the weighted average parallax of neighboring pairs is computed. If the parallax difference
between this pair and weighted average one is larger than a certain threshold, this point pair is considered as an

                                         ASPRS 2008 Annual Conference
                                   Portland, Oregon April 28 - May 2, 2008
outlier. The weight is determined according to the distance between this pair and the neighboring one. The larger is
the distance, the smaller is the weight.


                                               EXPERIMENTS
     The approach presented in this paper is tested with coaxial vehicle-based images provided by Toyota
MapMaster Incorporated. The image size is 1920 × 1080 pixels.
     As Fig.5, there are much more potential corresponding points on the buildings than those on the road and
moreover it makes no sense to match corresponding points in the sky. Therefore, only the areas including buildings
instead of the whole raw image are selected to generate epipolar images. The areas I and II to generate epipolar
images are illustrated as Fig.5. According to the area determined, the epipolar images were generated as Fig.6. In
Fig.6, although the straight lines in raw image become arcs in epipolar image, the point features are still apparently
clear. Moreover, the maximum vertical parallax in the epipolar images is only 1.317 pixels.




                                                                  II                                I
                II                                I




                                     Figure 5. The areas to generate epipolar images.

    The matching algorithm presented in section 3 was employed to match corresponding points in both of raw
images and epipolar images. The matching results are as Fig.5 and 6 respectively. As Tab.1, in the area I and II, only
10 and 25 corresponding point pairs were matched from raw images respectively, however the numbers of
corresponding point pairs is respectively 244 and 365 in epipolar images. Compared to the matching in raw images,
the matching in epipolar images not only simplifies the process, but also increases the numbers of corresponding
point pairs matched over 10 times.




                                                 Figure 6. Matching results.




                                         ASPRS 2008 Annual Conference
                                   Portland, Oregon April 28 - May 2, 2008
                         Table 1. The numbers of corresponding point pairs matched

                                           In raw images   In epipolar images
                                      I          10               244
                                      II         25               365


                                               CONCLUSIONS
     Epipolar image generation and corresponding point matching for coaxial stereo pair was discussed in this paper.
As coaxial stereo pair as be concerned, the epipole is the intersection of baseline S1S2 and image. In this case,
epipoles lie in the image near the principal point so that the epipolar lines in images along the optical axis are
arranged in the direction of radial. As a result, the epipolar image generated by rearranging the epipolar lines is
distinctly different from the raw one. Although the straight lines in raw image become arcs in epipolar image, the
point features are still apparently clear and possible for 1D corresponding point matching. Moreover, compared to
the matching in the raw image, corresponding point matching in the epipolar image not only simplifies the process,
but also improve the success rate of matching.


                                                REFERENCES
Gluckman, J. M., Thoresz, K., and Nayar, S. K., 1998. Real time panoramic stereo. Image Understanding Workshop.
Harris, C. and Stephens, M., 1988. A Combined Corner and Edge Detector, Proc. Alvey Vision Conf., Univ.
         Manchester, pp. 147-151.
Kraus, K, Jansa, J., and Kager, H., 1997. Photogrammetry, Vol.2, Advanced Methods and Applications,
         Dümmler/Bonn, ISBN 3-427-78684-6.
Li, R., F. Ma, F. Xu, L. H. Matthies, C. F. Olson, and R. E. Arvidson, 2002. Localization of Mars rovers using
         descent and surface-based image data, J. Geophys. Res., 107(E11), 8004, doi:10.1029/2000JE001443.
Lin, S., Bajcsy, R., 2003. High Resolution Catadioptric Omni-Directional Stereo Sensor for Robot Vision, IEEE
         2003 International Conference on Robotics and Automation.12-17 May 2003 (Taipei, China).
Moravec, H.P., 1977. Towards Automatic Visual Obstacle Avoidance, Proc. 5th International Joint Conference on
         Artificial Intelligence, pp. 584.
Wang, Z., 1990. Principles of Photogrammetry, Publishing House of Surveying and Mapping, Beijing, pp. 27-29.
Yuille, A. L. and Poggio, T. A., 1986. Scaling theorems for zero crossings, IEEE Transactions on Pattern Analysis
         and Machine Intelligence, 8(1): 15-25.
Zhang Z., Kang Z., 2004. Several Issues on Photogrammetry based on forward moving images along the optical
         axis, The 4th International Symposium on Mobile Mapping Technology (MMT’2004), Kunming, China,
         29-31 March 2004.
Zhang, Z, Zhang J. (1997). Digital Photogrammetry, The Press of Wuhan University. pp. 114-115.




                                        ASPRS 2008 Annual Conference
                                  Portland, Oregon April 28 - May 2, 2008

								
To top