Center for Advanced Sensors David J. Russomanno, Ph.D. Professor and Chair Department of Electrical and Computer Engineering Herff College of Engineering The University of Memphis Knowledge Fusion Workshop December 1, 2005 Annapolis, MD Center for Advanced Sensors (CAS) NVL (NVESD) ONR Army Research Lab University of Memphis VU UAH Outline • CAS Background • CAS Projects Overview • Imaging Sensors Research • Knowledge Engineering Research Center for Advanced Sensors • Established in April 2005 via a cooperative agreement between ARL/ARO and The University of Memphis • Enhanced by over 25 years of collaboration between The University of Memphis and the U.S. Army’s NVESD, Redstone Technical Test Center (RTTC), and ARL • PIs reside in Electrical and Computer Engineering Electrical & Computer Engineering • 120 EE and 70 CpE undergraduate students 38% African-American • 50 MS and 15 PhD graduate students increasing efforts to recruit US citizens into graduate programs • 11 Faculty, 1 Post-Doc, 3 Adjuncts 8 research-oriented faculty 3 teaching- and service-oriented faculty ongoing search for tenure-track Assistant/Associate Professor • Research Focus Areas: imaging sensors and electronic devices – ARL/ARO, NVESD, ONR, EOIR, ERC biomedical imaging and devices – Whitaker Foundation, American Heart Association intelligent information systems – NSF • Anticipate $2M in FY 05-06 research expenditures Center for Advanced Sensors Memphis Sensor Simulation Know ledge Test-Bed Engineering Sensor Bio-Chips Architectures Nano- Wavefront Technology Sensing/Control Microstructure Lightw eight Fabrication Optical Systems Sensor Testing/ Diamond Calibration Technology UAH VU Center for Advanced Sensors Projects at The U of M • Performance Modeling with Image Processing Enhancements – Halford, Robinson Establish perception laboratory and provide support for models Establish methodology for incorporating image processing techniques into existing performance models Modify/adapt existing sensor models to account for image fusion Model performance enhancement by anomaly detection ATRs • Performance Modeling of Advanced Architecture Systems – Griffin Model short-to-mid range transmission in the THz Model antenna coupling Model optics and interaction with focal plane coupling system • Knowledge Engineering – Russomanno Sensor ontology development Ontology-driven algorithms Application Development Center for Advanced Sensors Projects at VU and UAH • VU – Bio-Optics of Vision, IR Display, Bio-Optic Sensor Electrodes – Bonds, Davidson Research natural (living organism) visual sensory representation as performed by neural assemblies, recording isolated neural activity across a network of cells Determine the coding of visual signals by cell populations and achieve a working view of the neural code to achieve superior electronic approaches to rapid AI night vision image signal processing Engages multiple disciplines, including (1) visual neuroscience,(2) imaging, processing and detection, (3) automated decision making/classification and (4) statistical analysis • UAH – Wavefront Sensors – Reardon Existing wavefront sensors tend to be computationally intensive Desirable to reduce or eliminate computational steps Research utilizes a purely optical means of decomposing the wavefront into known forms Imaging Sensor Modeling Detector Display Array & Optics Cooler Human Vision Scanner Electronics ATR Atmosphere Visible – Sunlight (Digital Video) SWIR – Laser Illuminated Infrared – “Thermal Glow” (FLIR) Target and Background Submillimeter – “Molecular Glow” (THz) Millimeter & Longer – Obstruction (RADAR) Role of Models Test & Evaluation Sensor M odel M odel M odel Concepts Development Validation Applications Design Technology Field Test Trades Push Data Component HITL Perception Training Technology Experiments Testing models 1 0.8 Theory / 0.6 Literature 0.4 War Gaming 0.2 0 0 10 20 30 Acquisition Decisions Product Improvements Sensor Performance M odels - NV-Therm, Acquire, Search, I2, TV, Laser Basic Applied Development Single Device Production Research Research Development & Fielding CAS Perception Laboratory Indigo Merlin LWIR and MWIR cameras LightSpace 1024Z 3 Dimensional Monitor Large set of Military visible, LW, and MW images Synthetic Image Generation Effects of band-limited noise, blur on minimum contrast for ID • Perception tests performed include: TOD (Triangle Orientation Description) ID of military vehicles • Assessment of perception as a function of range for imaging sensors in various environments TOD Vehicle ID Perception Lab Experiments Urban Operations and AT/FP • Provide the sensor design and analysis models for the military user, system designer, and the war gamer in the urban environment [TV, I2, SWIR, MWIR, and LWIR] Two Main Areas: Search and Night Imagery Day Imagery Target ID MWIR LWIR NVESD Twelve Target Set Phenomenology & Radiometry • Objective: Provide state-of-the-art radiometric field measurements to the NVESD research community and other government agencies Phenomenology • Extensively used in sensor modeling and development • Not significantly exploited once sensor is deployed • Incorporate aspects of phenomenology into sensor ontologies to enhance confidence in acquired sensor data? Source of Background/Context Knowledge Speckle Simulation Electric Field Intensity Transfer Power Transfer y random EBCCD E1 Pe j Collection Optics x E ' ( x, y ) E ( x, y ) * *h( x, y ) P' ( x, y ) | E ( x, y ) | ' 2 Field Results & Simulation Turbulence Field Results Third Generation FLIR Modeling • Objective: To provide sensor designers, analysts, and war gamers with a physics-based, multi-spectral modeling capability in support of the next generation of FLIR sensors Band 1 Band 2 Fused Image Anomaly Detection ATR Input Anomaly Detection ATR Output THz Imagers – Concealed Weapons Detection Knowledge Engineering Activity Constellations of heterogeneous sensors Vast set of users and applications Airborne Network Services Sensor Web Enablement Weather Surveillance Application Services • Distributed self-describing sensors and related services • Link sensors to network and network- centric services Chemical Biological • Common encodings, information models, Detectors Detectors and metadata for sensors and observations • Access observation data for value added processing and decision support applications Sea State • Users on workstations, web browsers, and mobile devices Ontologies (Prior Work in implementing Plausible Inference* on the Semantic Web) ow nedBy partOf ow nedBy Car Grover Trunk Car Car Grover partOf containedIn containedIn ownedBy containedIn neither Wheel SpareTire Ernie Rj We implemented, using Semantic Web infrastructure, a symbolic algorithm that Ri reads/parses RDF and determines when the ? composition of Ri and Rj is nonempty and whether it is a subset of Ri or Rj or neither. *see [Huhns, 89] [Russomanno, 03], [Russomanno, 06] Requires Richer Relation Semantics <!-- Definition of causedBy property --> <rdf:Property rdf:ID="http://www.ee.memphis.edu/ksl/uofM_eece#causedBy"> <rdfs:comment>causedBy(Event1, Event2) means Event1 is caused by Event2. It is the property used for object-object event causation. Note that value "no" has been asserted for temporal because the range element precedes the domain element.</rdfs:comment> <rdfs:domain rdf:resource="http://www.ee.memphis.edu/ksl/uofM_eece#Event"/> <rdfs:range rdf:resource="http://www.ee.memphis.edu/ksl/uofM_eece#Event"/> <uofM_eece:composable>yes</uofM_eece:composable> <uofM_eece:connected>yes/no</uofM_eece:connected> <uofM_eece:functional>yes</uofM_eece:functional> <uofM_eece:homeomerous>no</uofM_eece:homeomerous> <uofM_eece:intangible>n/a</uofM_eece:intangible> <uofM_eece:intrinsic>yes/no</uofM_eece:intrinsic> <uofM_eece:near>n/a</uofM_eece:near> <uofM_eece:separable>yes</uofM_eece:separable> <uofM_eece:structural>n/a</uofM_eece:structural> <uofM_eece:temporal>no</uofM_eece:temporal> </rdf:Property> CAS Knowledge Engineering • Activity: Build an ontology-based, knowledge repository of representative sensors Logical data model analysis and ontology design Population of an ontology with sensor instances Construction of Ubiquitous Sensing Prototype Test Bed • Objective: Tap in-house sensor expertise to develop knowledge models Ubiquitous Sensing Environment Connecting to infrastructure Discovering, extracting, collecting, and sharing data ... Sensor Node S Base Station S S S M Other Sensors, M M M M S … Sensor Netw orks, and/or Gatew ay: Bridges the sensor G E N netw ork to another Other Intelligence Sensors: S netw ork or platform S Sources Acoustic O Temperature R M Light Mote: S S Radiation N Microcontroller Vibration E Dynamic and Heterogeneous Environment Mesh Netw orking S/W Low Pow er RF transceiver M M Chemical Biological S T W Battery/solar pow ered ... O M R K CAS-Russomano et al.: S S S M M M 1. How do we expose sensor data, meta data, capabilities, and S services within a network-centric environment? ● Raw byte streams? Semantic Web? M 2. How do we discover and process data from multiple sensor sources for various tasks? ● Need explicit declarations and search strategies 3. How do we utilize sensors and other info sources for automated reasoning? ● Need shared understanding of semantics 4. How do we distribute tasks and processing across the network? ● Must handle vast numbers and varieties of data sources, adaptive networks and applications Ontologies needed for remote sensing through In-situ sensors U of M’s OntoSensor* • Work in progress … attempting to take advantage of local expertise in “traditional” physics-based sensor models and phenomenology • OntoSensor defines a set of concepts, taxonomies and relations common to many sensors • Seeks to leverage Semantic Web infrastructure: OWL, OWL-S, Rules, etc. • Create sensor profiles that commit to OntoSensor with attributes, properties, and services • Deploy aspects of OntoSensor in applications ASAP Incrementally develop “deeper” knowledge models as familiarity with physics-based sensor models evolves * see [Russomanno, 05a] [Russomanno, 05b] OntoSensor Leverages Existing Work • Sensor Markup Language (SensorML) Open Geospatial Consortium Initiative U of M is a (non-voting) member of the OGC Sensor Ontologists should be aware of this effort Developed from a Software Engineering rather than a Knowledge Engineering Perspective Not an ontology (UML models with XML realization only) No formal semantics Loose generic model May provide a good organizational framework within which an ontology can be created OGC revisions are ongoing (watching from the sidelines … incorporating aspects of SensorML into OntoSensor as appropriate) SensorML Conceptual Models* Possible SensorML Applications Include: • Coincident search for relevant data (i.e., data discovery) • On-demand processing of data products • Dynamic on-demand fusion of disparate sensor data • Pre-mission planning • Onboard applications • Autonomous operation/target recognition • SensorWeb communications of location and targets • Direct transmission of data and processing information to remote sites • Determination of sensor footprint with time • Intelligent retrieval and co-registration of sensor data • Visual fusion of disparate sensor data *SensorML material from [OGC 04-019, 2004] Identifier & Classifier • Identifier includes a type definition (e.g. shortName, longName, serialNumber, noradID, missionID, etc.) and a codeSpace which takes a URI. The codeSpace identifies the authority source (typically an online Dictionary or Registry) for the value. • Classifier object provides a means of providing several classification tags with the description. For instance, a single sensor might be classified as “remote observing”, “infrared detector”, “airborne”, “civilian”, and “atmosphere observing”. Such classifications could assist in sensor discovery. SensorML Capabilities • hasCapabilities property provides information that might be useful for sensor discovery • includes properties for supportedApplication, performanceProperties, and taskableProperty SensorML Interface • Sensors might support the IEEE-P1451 interface for low-level data transfer, or perhaps a specific Web service interface for tasking the sensor or retrieving data LocationModel • Each component, platform, sensor, and sample has its own local CRS (specified by the hasCRS property), which must ultimately be related to some geodetic CRS (e.g. latitude, longitude, altitude). The process by which this occurs is dependent on the LocationModel used. SensorML Parameters • Parameters are used within various SensorML classes as values for various properties of the type, dataComponent, as well as physicalPropertyTypes. • Example properties include latitude, speed, stepAngle, wavelength, or timeStamp. SensorML Response Model • ResponseModel is a particular type of ProcessModel that describes the sensor’s response to some phenomenon • Describes the process by which a sensor measures a Phenomenon and converts that observation to an output Product • ResponseModel provides information regarding the sensor’s sensitivity to a phenomenon and the quality of its measurements Logical Data/Knowledge Model IEEE SUMO U of M’s OntoSensor OntoSensor effort includes Agent Shell Development • Objectives Locate sensors via intelligent search Query specs/capabilities of a sensor and/or network Task sensors via abstract service specifications • Goal: Achieve these objectives at the agent level without the agent having to deal with proprietary software designed for each individual sensor • Important for adaptive fusion algorithms without a priori knowledge of specific sensors Summary of Accomplishments through 2 nd Quarter • Logical Data Model Analysis: Analyzed the 1.0.0 (beta) specification of the Sensor Model Language (SensorML) for In-situ and and Remote Sensors published by the Open GIS Consortium (OGC). • Logical Data Model Design: Started implementation of a prototype sensor ontology (OntoSensor) using the SensorML specification (in-part), IEEE Suggested Upper Merged Ontology (SUMO), International Organization for Standardization 19115, and constructs of the Web Ontology Language (OWL). • Sensor Repository Population: Skeletal data/knowledge about several sensors and motes have been instantiated. • Implementation: Coded initial version of Semantic Web Expert System Shell (SWEXSYS) capable of querying OWL knowledge bases. Includes Dempster-Shafer, voting fusion and other elementary fusion algorithms. • Prototype Construction: Started design and configuration of ubiquitous sensing prototype test bed. … 433 MHz TBD Prototype Construction (CAS-Memphis) • Sensor nodes are built using low-cost Sensor Instances and/or magnetic, passive infrared, light, Semantic Web LAN Aggregations (in OWL) temperature, humidity, barometric Expert System Commit to Sensor Node S pressure, and acoustic sensors provided Shell (SWEXSYS) ES 223 OntoSensor Base Station S S S M by Crossbow Inc., and others M M M M S E • Wireless software provided by Crossbow’s G N S mesh networking protocol S O R • Mote packets are stored in a Postgress OntoSensor Know lege Base M N database for subsequent extract to (OWL) ES 231 S S E Semantic Web data repositories 916 MHz T M M W • MWIR and LWIR cameras from FLIR, Inc. S O R • M K SWEXSYS can read sensor repositories in the OWL knowledge base S S S 1 • Possible SWEXSYS Applications: Sensor M M M Instances and/or Aggregations Thermal camera calibration using (in OWL) Commit to Sensor Node S wireless MEMS sensors (in situ) in OntoSensor Base Station S S S M field of view for ground truth M M M M S E Map pixel to GPS location using G N S wireless MEMS sensors (in situ) in S O R field of view for ground truth M N Ontology-driven algorithms to specify S S 2.4GHz E T sensor states based on information M M W from base stations, perceived S O R environment and overall objective of M K agent 2 Increase confidence in target identification via incrementally acquired evidence and/or data/knowledge fusion Typical specs included in sensor repository that commits to OntoSensor Knowledge Engineering 3rd Quarter Plans • Add more detail about sensors to ontology (emphasize low-cost sensors that comprise the sensor networks first, followed by more sophisticated imaging sensors) Sensor provides Service presents described by supports • Increase number of sensor Service Service Service Profile Model Grounding instances/repositories that commit to What the service does How it w orks How to access it OntoSensor in a network-centric environment • Explore capturing sensor services into the ontology using OWL-S Application Development (ASAP) • Support ARL ongoing application development • Persistent Threat Detection System (PTDS) sensor markup using OntoSensor MWIR, Color Video, B/W Video, Acoustic Sensor, etc., suite on single platform • PTDS interfaces to Counter Rocket and Mortar (CRAM) system • Project: Implement PTDS meta data spigot & ad- hoc query support using OntoSensor concepts to support future interoperability and possible fusion CAS & Affiliated Personnel • CAS-Memphis • ARL Carl Halford, CAS Director Steve Murrill, CAS Program Manager Steve Griffin Gary Wood Aaron Robinson Bill Ruff David Russomanno Ronnie Sartain • CAS-VU Greg Sztankay A.B. Bonds • NVESD Jim Davidson Ron Driggers • CAS-UAH Eddie Jacobs Pat Reardon Phil Richardson Joe Geary • ONR Keith Krapels References 1. [OGC 04-019, 2004] Open GIS Consortium Inc. (2004) “Sensor Model Language (SensorML) for In-situ and and Remote Sensors,” Version 1.0.0 (beta), M. Botts (Editor). 2. [Huhns, 89] M. Huhns and L. Stephens (1989) “Plausible Inferencing Using Extended Composition,” Proceedings of the Eleventh International Joint Conference on Artificial Intelligence, Detroit, MI, pp. 1420-1425. 3. [Russomanno, 03] D.J. Russomanno and C. Kothari (2003) “An Implementation of Plausible Inference for the Semantic Web,” The International Conference on Information and Knowledge Engineering (IKE03), CSREA Press, Las Vegas, Nevada, pp. 246-251. 4. [Russomanno, 06] D.J. Russomanno (to appear in 2006) “A Plausible Inference Framework for the Semantic Web,” Journal of Intelligent Information Systems. 5. [Russomanno, 05a] D.J. Russomanno, C. Kothari and O. Thomas (2005) “Sensor Ontologies: From Shallow to Deep Models,” The 37th Southeastern Symposium on Systems Theory, IEEE Press, Tuskegee, Alabama, pp. 107-112. 6. [Russomanno, 05b] D.J. Russomanno, C. Kothari and O. Thomas (2005) “Building a Sensor Ontology: A Practical Approach Leveraging ISO and OGC Models,” The 2005 International Conference on Artificial Intelligence, CSREA Press, Las Vegas, Nevada, pp. 637-643.