Stereoscopic Reconstruction of Flight Paths of Foraging Bats Using Multiple Infrared Cameras
Lisa B. Premerlani, Margrit Betke, John J. Magee, Stan Sclaroff, Jonathan Reichard, Nickolay Hristov, Thomas H. Kunz Boston University, Boston, MA
Computer Science
Problem Definition
Computer Science
Given two thermal videos of the same bat, reconstruct the bat’s three dimensional (3D) trajectory.
To study the foraging behavior of bats, including the Brazilian free-tailed bat (Tadarida brasiliensis), we took infrared (IR) thermal videos of bats during warm weather nights in southcentral Texas and used computer vision techniques to detect and track the bats. We analyzed foraging activity over a small body of water that was adjacent to a corn field from which corn earworm moths were emerging. Two cameras, approximately 20 meters apart, were used to record simultaneous video images for the purpose of producing a stereo reconstruction of 3Dimensional (3D) trajectories of the bats.
Methods
Computer Science
• Temporal calibration of videos • Spatial calibration of cameras
– Used Horn’s Relative Orientation algorithm – Used homemade calibration devices to obtain corresponding points
• Triangulation to obtain 3D bat coordinate in each frame
Spatial Calibration
Computer Science
• •
Obtain a set of n corresponding points from left and right images (l, r) Horn’s method: An iterative least squares method that solves the linear system: A BT qT 0T B C 0T dT q 0 0 0 0 d 0 0 d q 0 0
δq
δd λ μ ν
= -
s
t 0 0 0
dT
qT
0
0
0
The upper 8-by-8 part of the matrix is ∑i=1..n wiciciT where ci = ridli* ri*qli
and wi are weights.
The right side of the equation is given by ∑i=1..n wieici where ei = rid · qli q is the quaternion that represents the rotation that registers the two cameras dq* is the quaternion that represents the translation that registers the two cameras
Obtain Corresponding Points
Computer Science
Hand-made calibration devices were rotated in scene. Over time this generated many sets of corresponding points.
Calibration Device 1
Calibration Device 2
Triangulation
Computer Science 3D Position of Bat
Camera 1
Camera 2
Once the relative location of the cameras is known, a ray can be projected from each camera’s center of projection through the image of the bat in the image plane. The point of intersection of these two rays is the 3D position of the bat.
Triangulation
Computer Science
Once the rotation (R) and translation (t) that registers the cameras is known, the z coordinate of a bat at (xl', yl') in the left image and (xr', yr') in the right image can be computed: (R11xl' / f + R12yl' / f + R13)zl + t1 = zrxr' / f (R21xl' / f + R22yl' / f + R23)zl + t2 = zryr' / f
(R31xl' / f + R32yl' / f + R33)zl + t3 = zr
zr is the bat’s z coordinate in the coordinate system of the right camera zl is the bat’s z coordinate in the coordinate system of the left camera
f is the focal length of the cameras
Results
Computer Science
2D trajectory measured by left camera
2D trajectory measured by right camera
3D reconstructed view
Conclusions
Computer Science
• Calibration devices, designed by our team, were effective
• Horn’s relative orientation method aided in camera calibration • Computer vision techniques can help biologists obtain 3D trajectories of foraging bats • 3D reconstruction can be done on field data, provided proper citing of cameras and calibration devices
Future Work
Computer Science
• • •
Obtain 3D reconstruction using three-camera setup Perform simultaneous 3D reconstruction on many bats Analysis of 3D databases of foraging bats can provide valuable information on:
Turning radius of foraging bats Proximity of foraging bats to one another Proximity of bats to objects in their flight paths Flight trajectories of bats in different habitats Flight trajectories of different bat species