PowerPoint Presentation - Hard Science Questions, Software solutions

							Video Annotation and
  Reference System
       (VARS)
     Brian Schlining
                   About Me
Software Engineer
Specialize in Data Systems and Data Analysis

Bachelor of Science, Biology.
University of Maryland, College Park

Masters of Science, Physical Oceanography.
Moss Landing Marine Laboratories.
             Overview
The problem of managing video data
What is VARS?
History of VARS development
VARS Deployment at MBARI
Example of VARS data analysis
ROV Ventana

   Maximum depth: 1800 meters (5,906 feet)




                        QuickTime™ and a
                 Sore nson Video decompressor
                 are neede d to see this picture.
ROV Tiburon

   Maximum depth: 4000 meters (13,123 feet)




                       QuickTime™ and a
                Sore nson Video decompressor
                are neede d to see this picture.
           ROV Video
Video Cameras




                         QuickTime™ and a
                  Sore nson Video decompressor
                  are neede d to see this picture.
                                            ROV Video
• More than 300 ROV dives per year
• More than 12,000 hours of video observations
generated in 16 years (that’s a stack of video
tape nearly 1/2 mile tall.)



                QuickTime™ and a
         Sore nson Video decompressor
         are neede d to see this picture.
              The problem
Q: How do we use the video data to produce
  good science?

A: A picture is worth a thousand words, if you
  can find it.
A: Use video for quantitative and qualitative
  science.
       The solution…annotations
• Trained science staff identify animal taxonomy, animal
behavior, geological features, evidence of human impact, and
other objects recorded in the ROV video stream
• Annotations are stored in a database shared between MBARI
scientists.
• 1,750,000 annotations
      What is an annotation?
• Identification of an object of interest
• The ID of the tape
• The time-code on the tape
        What is an annotation?




Cranchiidae at 02:25:30:28 on Tape T0325-03
        What is an annotation?




Cranchiidae at 02:25:30:28 on Tape T0325-03
      time:         2002-12-21 17:02:57
      latitude:     36.331
      longitude:    -122.901
      depth:        984.8 m
      salinity:     34.383 psu
      temperature: 3.940 C
         What is an annotation?




Cranchiidae at 02:25:30:28 on Tape T0325-03
      surface-color | self | red
      swimming | nil | nil
                                      Descriptive information
      image-quality | self | good
         What is an annotation?




Cranchiidae at 02:25:30:28 on Tape T0325-03
      surface-color | self | red
                                      More than 1 observation
      swimming | nil | nil            per frame is allowed
      image-quality | self | good
Nanomia
      eating | krill | nil             Can link observations
What do you call something?
What do you call something?



               Cockatoo-squid



            Whoa…that’s a common name.

            Need to reference a scientific name
What do you call something?



               Cranchiidae (Cockatoo-squid)



            Much better…but who can remember
            how to spell Cranchiidae?
What do you call something?
             Mollusca
             Cephalopoda


              Squid

               Cranchiidae (Cockatoo-squid)



            Need to be able to reference phylogeney
What do you call something?
            Eukaryota
            Animalia (Metazoa)
             Mollusca
             Cephalopoda
              Coleoidea
              Decapodiformes
               Teuthoidea (Squid, Teuthida, Theudoidea)
                Oegopsida
                Cranchiidae (Cockatoo-squid)
                 Galitheuthis
                  Galitheuthis phyllura
   VARS
  A suite of 3 Java applications


Knowledgebase

Annotation

Query
   VARS Knowledgebase
  Manages constraint lexicon and references


Knowledgebase

Annotation

Query
   VARS Annotation
  Generates video-annotations and frame-grabs


Knowledgebase

Annotation

Query
   VARS Query
  Searches for and retrieves information and frame-grabs


Knowledgebase

Annotation

Query
  VARS History
1st generation – Free Text         •Inconsistent annotations
                                   •Difficult to search
2nd generation – VICKI/VIMS
  Constrained annotations
  Written in Smalltalk
  Generated files

3rd generation – VARS
  Constrained annotations
  Written in Java
  Writes information directly to database
  VARS History
1st generation – Free Text

2nd generation – VICKI/VIMS
  Constrained annotations
  Written in Smalltalk            •Difficult to modify and maintain
  Generated files                 •Requires file management
                                   •Difficult to change annotations

3rd generation – VARS
  Constrained annotations
  Written in Java
  Writes information directly to database
  VARS History
                                 Why Java?
1st generation – Free Text       •Cross platform
                                 •Good database API (JDBC)
                                 •Good pool of developers
2nd generation – VICKI/VIMS
  Constrained annotations       Why use a database?
  Written in Smalltalk          •Easy to manage data
                                 •Changes are instantly available
  Generated files               •Easy to make changes or update
                                  errors
3rd generation – VARS
  Constrained annotations
  Written in Java
  Writes information directly to database
VARS Deployment
VARS Deployment




            Workstation
VARS Deployment




          Database Server
VARS Deployment




            VCR
VARS Deployment




         Database components
VARS Deployment




          Timecode capture
            VCR Control
VARS Deployment




           Frame capture
VARS Deployment
     Example - Seep Search
Search for ‘cold seeps’
  ‘Cold seep’ = Source of H2S, CH4
  ‘Cold seep’ = Chemosynthetic biological community
  Visual imaging is required to identify seeps
  Use indicator species: Vesicomyid Clams,
   Vestimentifern worms, bacterial-mats, seeps
     Example - Seep Search
Q: Where do seeps occur and why?

A: Common wisdom:
  Seeps are associated with faults.
  Is this true in Monterey Bay?
Example - Seep Search



             QuickTime™ and a
      Sore nson Video decompressor
      are neede d to see this picture.
Example - Seep Search



             QuickTime™ and a
      Sore nson Video decompressor
      are neede d to see this picture.
1999
Example - Seep Search




                        2003
2003
2003
2003
Example - Seep Search
      Example - Seep Search
Q: Where do seeps occur and why?

A: In Monterey Bay, seeps are associated with
  steep slopes and slide scars. (i.e. areas of
  recent erosion)