Bio-inspired Learning for a
Colony of Microsystems
Who am I?
• A computer vision researcher
• Interested in representation and
• Over the past 10 years heavy involvement
with understanding video sequences
• A participant in the ARO MURI on Micro
• Collaborations with Prof. M. Srinivasan.
Pattern Recognition and Bees
• Behavior analysis of insects has led to advances in
navigation, control systems etc.
• Goal: To automate the tracking and labeling of insect
i.e., track the position and the behavior of insects.
• Joint work with Prof. M. Sreenivasan of ANU
• To appear in IEEE Trans. PAMI
• All insects have similar anatomy.
• Hard Exoskeleton, soft interior.
• Three major body parts- Head, Thorax
• Each body part modeled as an ellipse.
• Anatomical modeling ensures
– Physical limits of body parts are
– Accounts for structural limitations.
– Accounts for correlation among
orientation of body parts
– Insects move in the direction of
Waggle Dance- 1
• Orientation of waggle axis Direction of Food
source.(with respect to sun).
• Intensity of waggle dance Sweetness of food
• Frequency of waggle Distance of food source.
• Parameters of interest in the waggle dance
– Waggle Axis : Average orientation of Thorax during Waggle.
– Duration of Waggle : Number of frames of waggle in each
segment of the dance.
Mixture Modeling for Behaviors
• Low level Motion states
– Straight, motionless, waggle,
• Each Behavior is a Markov
model on such motion states.
• Switching between behaviors
is modeled as another Markov
• Detect frames of waggle dance
by looking at
– Rate of change of Abdomen
– Average absolute motion of
center of abdomen in the
direction perpendicular to the
axis of the bee.
Shape, Motion and Behavior
• Tracking using a particle filter.
• Behavioral model in addition to motion model in
the normal particle filter framework.
• Track both position and orientation of various body
parts and the behavior exhibited by the bee.
• Observation model:
– Mixture of Gaussians.
– 5 Exemplars for the appearance of the bee.
• Maximum Likelihood estimate for both position
The Grand Challenge - 1
• More and more MAVs and Microrobots will be employed for a variety
• Microsystems/pupil ratio will increase at an alarming rate.
• We need to figure out how a colony of such systems (less than 100)
can organize themselves for carrying out a few well-defined tasks.
– Landmark-based navigation
– Terminal guidance
– Looking for anomalies
• Much is known about how honeybees carry out tasks
related to navigation, perching, hunting for honey, etc.
– Prof. Srinivasan and several others
The Grand Challenge – 2
• Examples of problems to be studied.
• Sensors for MAVs and Microrobots
• How to keep track of other microsystems in the colony
– Tracking/Tagging a large number of movers in a restricted space
– Bees are always on the move. Microsystems need to move only when
– Keeping an account of who went out and who came in
– How to organize a sub-group of micro-systems for carrying out a
– Activity recognition
– Waggle dancing by MAVs!
• How to nourish microsystems?
– Power, communication issues
– Self-evaluation of their well being
– Call for help.
• How will humans interact/interface with the colony?