Learning Center
Plans & pricing Sign in
Sign Out



									Situational Awareness in Emergency Response

            Dr. Sharad Mehrotra
       Professor of Computer Science
         Director, RESCUE Project

                     Crisis Response
• A massive, multi-organization operation                                 LEVELS       LAW

• Many layers of government                              FEDERAL
      Federal: FEMA, FBI, CDC, national
       guard, ..
      State: Governor’s Office of Emergency               STATE
       Services (OES), highway patrol, …
      County: county EOC, police, fire                                     RESOURCE COORD
       personnel, …                                        LOCAL
      City: city emergency offices, police,                                       OPERATIONS
       firefighters, …                                     EMC C2

                                                    Incident command C2
• Volunteer Organizations
      Red cross, organized citizen teams            FIRST RESPONDERS

• Industry                                                 VICTIMS
      Gas, electric utilities, telecommunication
       companies, hospitals, transportation
       companies, media companies ….
                                                                          Los Angeles County
                                                                        Emergency Management

                                                                                    LA County Emergency
                                                                                    Management Council
                          Board of Supervisors

                           Chair of the Board
                      Operational Area Coordinator

                                                              Director of Emergency Operations

                                                           Center              Other Entities

Disaster Management
Area Coordinators
                                        Sheriff Contact         Emergency Mgmt
                                           Stations            Information System

                                         Cities of Los Angeles County (87)
                 Operational View of Response

• Crisis Management
     Field level operation
     Command and control
     Usually local government in-charge

• Consequence Management
     Gather information
       • Field, Cities, Special districts, County departments, Other EOC
     Analyze consequences with focus on the future
     Develop plan of action
       • Life safety, Property loss, Environment, Reconstruction
     Establish who is responsible
                Operations- Consequence Analysis

Potential need for:                        Public Safety
• Security for damaged/evacuated
• Route management
• Civil disturbance control
• Casualty/Fatality collection points
• Fire fighting/HAZMAT support

                                        • Shelter requirements
                                        • Impact on poor
                                        • Language, other cultural
                                        • Food/water distribution
                                        • Impact on schools
           Care/Shelter                 • Impact on non-profit
                 Operations – Consequence Analysis

• Need for building inspections     Construction
• Removal of hazardous materials
• Demolition/debris removal
• Transportation network – impact
  and restoration
                                       CONSTRUCTION & ENGINEERING
• Water/sewage/flood control
  system impacts

                                       •Impact of utility outages
          Logistics                    •Priorities for restoration
                                       •Impact on purchasing system
                                       •Impact on transportation
                                       •Priorities for transportation
                                       •Other support
               Role of Information in Response

Hypothesis: Right Information to the Right Person at the
 Right Time can result in dramatically better response

           Response                    Quality of
         Effectiveness                 Decisions
       • lives & property saved    • first responders
       • damage prevented          • consequence planners
       • cascades avoided          • public

           Quality &                   Awareness
         Timeliness of             • incidences
                                   • resources
          Information              • victims
                                   • needs
Challenges in Situational
              RESCUE Project

                                      Research Team
                                            • Privacy
The mission of RESCUE is to                 • Security
                                            • Trust

enhance the ability of emergency
                                            • Natural Hazards Center
response organizations to rapidly           • Social Science

   adapt and reconfigure crisis             • Data Management
                                            • Security and Trust
  response by empowering first                                       • Transporation Modeling
                                            • Disaster Analysis
                                            • Earthquake Engineering • Urban Planning
    responders with access to               • GIS
                                            • Civil Engineering         • Privacy
 accurate & actionable evolving             • Data Analysis & Mining • Social Science
                                            • Data Management           • Transportation Science

      situational awareness                 • Middleware & Distributed Systems
                                            • Civil Engineering
                                            • Transportation Engineering
                                            • Computer Vision              • Wireless
                                            • Networking
                                            • Multimodal Speech

     Funded by NSF through its large ITR program
                                   RESCUE Partners

                 Industrial Partners                                          Government Partners
                                                                        California Governor’s    California Governor’s
        5G Wireless
                                                AMD                     Office of Emergency       Office of Homeland
   Broad-ranged IEEE 802.11
                                          Compute Servers                      Services                 Security

      Apani Networks
                                   1st responder (LAPD), and threat      City of Champaign        City of Dana Point
    Data security at layer 2
                                           analysis software

  Community Advisory Board                                                  City of Irvine       City of Los Angeles
                                     Visualization equipment SDK

          Convera                    Cox Communications
                                                                           City of Ontario
     Software partnership              Broadcast video delivery                                   City of San Diego
                                                                          Fire Department
           D-Link                             Ether2                  Department of Health and
  Camera Equipment and SDK             Next-generation ethernet                                  Lawrence Livermore
                                                                      Human Services – Centers
                                                                        for Disease Control      National Laboratory
             IBM                          ImageCat, Inc.
Smart Surveillance Software (S3)   GIS loss estimation in emergency
 and 22 e330 xSeries servers                   response                                            National Science
                                                                        Los Angeles County
          Microsoft                         Printronix
           Software                       RFID Technology
                                                                                                 Orange County Fire
                                                                           Orange County
The School Broadcasting                                                                              Authority
                                     Vital Data Technology
                                         Software partnership
  School based dissemination
                                                                         U.S. Department of
      Walker Wireless                                                    Homeland Security
  People-counting technology
                                                                RESCUE Research
Security, Privacy& Trust

                                                             Social & Disaster Science                 Engineering & Transportation
                       Cross cutting issue at every level

                                                              context, model & understanding of
                                                                                                       validation platform for role of IT research
                                                            process, organizational structure, needs

                                                                      Information Centric Computing
                                                                  enhanced situational awareness from multimodal data

                                                                   Networking & Computing systems
                                                            Computing, communication, & storage systems under extreme situations
Situational Awareness Research

      Extraction,     Situational    Decn.
      synthesis,         Data       Support
    Interpretation   Management      Tools
• Multimodal multi-sensor signal processing
      Robustness to noise – noise affecting one modality may be
       independent of the others.
        • E.g., multimicrophe speech recognition with background noise
      Complementary information in different modalities – certain events
       easier to detect in some modalities than others. By combining
       modalities we can build systems that detect complex events
        •   E.g., Tracking   people is easier in video whereas speaker identification is easier
            in audio.

• Exploit semantics & context for signal interpretation
      Knowledge of domain can help interpret data, fill missing values,
                  Exploiting Semantics for
                  Situational Awareness
• How does the system obtain & represent semantics?
      User specified
        • Language for specification of semantics, expressibility, completeness
      learnt from data
        • expressibility, training set might not be available for supervised learning,
          noise in data may skew unsupervised learning
• Principled approach to exploiting semantics to interpret data
      Probabilistic models?
• Efficiency
      Most such problems are NP-hard
• Generalizability of the approach
      Can we design a generalized approach that can be used to work
       across diverse types of data and for diverse situational awareness
                  Event Detection from sensors
• 2300 Loop sensors in LA
  and OC
• Goal: Detect events such
  as “baseball game” from
  loop sensor count data.
• Semantics:
      Historical traffic data both
       during game night and non-
       game night
      Data is, however,

• Smyth et. al. -- TRBC 06,
  07, UAI 07
                       Detecting Unusual Events

            Ideal model
car count

            Baseline model
car count

            Unsupervised learning faces a “chicken and egg” dilemma (and others)
            Inference over Time

            Time t                               Time t+1
                         p                                     p
Time,                                Time,
 Day          Event                   Day           Event

        l                                    l

                                 a                                     a
              True      Sensor                      True      Sensor
             Count       State                     Count       State

        q                                    q

             Observed                              Observed
              Count                                 Count

Note how many hidden variables are in this model
Detecting Real Events: Baseball Games
 Total Number       Graphical             Baseline
 Of Predicted         Model                Model
    Events      Detection of the 76   Detection of the
                  known events        76 known events

    203             100.0%               86.8%

    186             100.0%               81.6%

    134             100.0%               72.4%

     98              98.7%               60.5%

   Remember: the model training is completely unsupervised,
           no ground truth is given to the model
                Entity Resolution Problem
     Raw Dataset                                                 Normalized Dataset
                                                                 (now can apply data analysis techniques)

                                                                       John Smith           Intel
       ...J. Smith ...

                                                                       Jane Smith            MIT

              MIT                                                             ...             ...

         Intel Inc.
          .. John Smith ...               Extraction

       .. Jane Smith ...                  duplicates, ...)
                                                             Attributed Relational Graph (ARG)

                                                                              John Smith
                                                                                           Jane Smith

    The problem:                                                                     MIT
      "J. Smith"                ?


    (for any objects, not only people)                           (nodes, edges can have labels)

TODS 2005, IQIS 05, SDM 05, JCDL 07, ICDE 07, DASFAA 07, TKDE 07
                           Two Most Common Entity-Resolution Challenges

Fuzzy lookup                               Fuzzy grouping
– reference disambiguation                 – group together object repre-
– match references to objects                sentations, that correspond
     – list of all objects is given          to the same object

        ...J. Smith ...


          Intel Inc.
           .. John Smith ...

       .. Jane Smith ...

13 January 2011 DASFAA 2007, Bangkok, Thailand                   20
Example of the problem: Disambiguating locations

         DASFAA 2007, Bangkok, Thailand   22
Web Disambiguation

                Music Composer

                Football Player

                 UCSD Professor


                Botany Professor @ Idaho
                  Context Attraction Principle (CAP)

                                                                 publication P1
 if         “J. Smith”

         reference r, made in the context of entity x, refers to an
          entity yj         John E. Smith
                             SSN = 123
         but, the description, provided by r, matches multiple
          entities: y1,…,yj,…,yN,
 then                    John E. Smith        Joe A. Smith        Jane Smith

         x and yj are likely to be more strongly connected to
          each other via chains of relationships
                 than x and yk (k = 1, 2, … , N; k  j).

Can be translated into a graph connectivity analysis which can be interpreted using a
probabilisitic interpretation.
                    Experiments: Quality (web

By Artiles, et al. in SIGIR’05   By Bekkerman & McCallum in WWW’05

GDF vs. Traditional (Robustness)

                    GDF vs. Context (Bhattarya &


Fp measure





                    Publications Dataset   Movies Dataset

                Semantics in IE

• Extracting relations from free / semi-
  structured text (slot-filling)

• Exploiting semantics in IE
     declaratively specified
       • Specified as (SQL) integrity constraints
             On the relation (s) to be extracted
     Learnt from data
       • Mine patterns and associations from the data
        Declarative Constraints
create table researcher-bios (
       name: person
       title: thing
       employer: organization
       employer-joined: date
       doctoral-degree: degree
       doctoral-degree-alma: organization
       doctoral-degree-date: date
       masters-degree: degree
       masters-degree-alma: organization
       masters-degree-date: date
       bachelors-degree: degree
       bachelors-degree-alma: organization
       bachelors-degree-date: date
       previous-employers: organization
       awards: thing

CHECK employer != doctoral-degree-alma
CHECK doctoral-degree-date > masters-degree-date
                       Pattern mining over data

                                  Top10    med unranked    in US OUT


                 PI     PD   MI      MD
       BI                                        Stanford CSU   Tsinghua

                  PI    PD   MI       MD
        BI                                                             1989   2002

                 PI     PD   MI       MD

• Represent data as graph (RDF)
• Mine interesting patterns
      Including “graph associations”
• Example above
      Mostly people who have a PhD degree from a school outside the US
       also have their bachelors degree from a school out side the US.
                       Constraints in Action
John Smith, PhD, UCI, 2000, MS, MIT, 1997, BS, UCI,

John Smith, PhD, MIT, 1997, MS, MIT, 2000, BS, UCI,

John Smith, PhD, MIT, 2000, MS, MIT, 1997, BS, UCI,


                                        1. Order of degree dates
                                        2. No “toggling” of schools

                                           John Smith, PhD, UCI, 2000, MS, MIT, 1997, BS, UCI,

                                           John Smith, PhD, MIT, 1997, MS, MIT, 2000, BS, UCI,

                                           John Smith, PhD, MIT, 2000, MS, MIT, 1997, BS, UCI,
                               Experimental Results:
                 accuracy (F-measure) against constraints

                                                               ATTRIBUTE LEVEL
                                                               CD1. All (CS) PhDs awarded after 1950
        0.7                                                    CD2. Current position is from among a fixed list
        0.6                                                    CD3. PhD awarded only by a PhD awarding school
        0.4                                                    TUPLE:
        0.3                                                    CT1. People do not “toggle” between schools
                                                               CT2. Dates of doctoral, masters, and bachelors
                                                               degrees are in order
                                                               CT3. People do not work at the same place they
          0                                                    graduate from
              none   S     T    CT1 CT2      CD1 CD2 CD3 CD4   CT4. More likely that the grad school is US and the
                                                               undergrad school is outside US (vs other way around)
                                                               CT5. The grad school rank is at least as good (or
                                                               better) than the undergrad school rank

   researcher-bios domain
       (upto) 300 training documents (Web bios)
       Test set > 2000 documents
   Use RAPIER + Schema (type) information as baseline
   Add several constraints
   Improvement in both precision and recall

• Language for specifying constraints.
• Principled approach to exploiting constraints/ patterns for extraction.
• Scalability/efficiency
       Naïve approach of enumerating all possible worlds leads to exponential
       Problem NP hard even with a single FD (e.g., Year  BestMovie)

                                        Possible “worlds” (exponential !!)
        Crash, 2005                                 Crash, 2005
        Crash, 2006                                 Million Dollar Baby, 2005
                                                    The Lord of the Rings, 2004
         Million Dollar Baby, 2005
                                        Crash, 2006
         The Lord of the Rings, 2004    Million Dollar Baby, 2005
         The Lord of the Rings, 2005    The Lord of the Rings, 2005

                                                                 Crash, 2006
                                                                 Million Dollar Baby, 2005
                                                                 The Lord of the Rings, 2004
• Situational Awareness research in RESCUE
     Event detection, extraction, and interpretation from multimodal sensor data
     Situational data management (R. Jain, S. Mehrotra)
     Tools for decision support (S. Mehrotra)

• Two approaches:
     Exploiting multimodal and multisensor input
       • Multimodal speech, multi-microphone recog.  B. Rao,
       • Speech enhanced video  M Trivedi
       • Bayesian framework for Multi-sensor event detection  P Smyth,

     Exploiting semantics for interpretation
       • Text, entity disambiguation  S Mehrotra
       • Sensor data  P Smyth
       • Dynamic recalibration of video based event detection system exploiting semantics
         [MMCN 08]  S. Mehrotra, N. Venkatasubramanian
       • Automated tagging of images using speech input exploiting context and
         semantics [Tech. Report 08]  S, Mehrotra
• Situational awareness applications requires techniques to translate raw
  multimodal signals into higher level events.

•   Extensive research on signal processing but much of it studies different
    modalities in isolation

• Multimodal event detection and exploiting semantics to interpret data is
  a promising direction.

• A principled, generalizable, and a comprehensive approach represents a
  major challenge and an opportunity.

• Situational awareness tools built on such tools could bring
  transformative changes to the ability of first responders and response
  organizations to respond to crisis.
                     Connection to Cyber SA
Most of this talk focussed on here.
Techniques could translate for cyber awareness.
Also, through monitoring physical systems they directly could impact cyber SA.

                 Physical                                   Cyber
                 systems                                   Systems
              Situational                                 Situational
              Awareness                                  Awareness
              Of physical                               Of underlying
               Systems                                  cyber systems
                        Awareness of state of physical system
                        helps gain cyber situational awareness
                        and vice versa. I.e., State of physical systems can
                        serve as sensors for cyber systems and vice versa

To top