의미 모델링
Elaborating Sensor Data using Temporal and Spatial
Commonsense Reasoning
+
Mining Models of Human Activities from the Web
지능 기반 시스템 응용
2006. 11. 민준기
Agenda
B. Morgan and P. Singh, M. Perkowitz, et al., “Mining
“Elaborating Sensor Data Models of Human Activities
using Temporal and Spatial from the Web,” WWW 2004.
Commonsense Reasoning,”
BSN 2006. Introduction
The Problem Space Proposed Technique
LifeNet : A First-Person Evaluation
Model
Summary and Future Work
The Plug Sensor Network
2
The Problem Space
Two distinct directions for research
Human-out (This paper)
Telephone
Technology-in (Much sensor network research)
Text messaging on cell phones
Three topics
LifeNet probabilistic human model
The Plug sensor network
An experimental design for evaluation of the LifeNet
learning method
3
LifeNet : A First-Person Model
First-person common-sense inference model
OpenMind Common Sense, ConceptNet, The PlaceLab data,
Honda’s indoor common sense data
Attempts to anticipate and predict what humans do in the
world
All of the reasoning in LifeNet is based on probabilistic
propositional logic
“I am washing my hair” before “my hair is clean”
4
The Plug Sensor Network
Using for both learning common sense and for
recognizing and predicting human behavior
Using this sensor network to monitor how individuals
interact with their physical environment
Nine sensor modalities: sound, vibration, brightness,
current, wall voltage, acceleration
5
Agenda
B. Morgan and P. Singh, M. Perkowitz, et al., “Mining
“Elaborating Sensor Data Models of Human Activities
using Temporal and Spatial from the Web,” WWW 2004.
Commonsense Reasoning,”
BSN 2006. Introduction
The Problem Space Proposed Technique
LifeNet : A First-Person Evaluation
Model
Summary and Future Work
The Plug Sensor Network
6
Introduction : Recognize Humans Activities
Applications include activity-based actuation
Dimming lights when a video is being watched
Providing directions for someone using unfamiliar facilities
etc.
Ubiquitous, proactive, disappearing computing
Computers have to understand people’s needs by observing
their physical activities (and to act autonomously)
The cost of developing recognition infrastructure is too high
Even small classes of activities is hard to recognize
A broadly applicable system should be general-purpose
and easy to use
7
Introduction
Motivation
Vision based systems
None have reported detecting more than tens of activities in
practice
The features robustly detectable from vision are coarse
Represent the relationships between “blobs” in the image
rather than specific objects
Each activity is expensive to model
Learning of the models
The developers define the structure of the possible models
System tunes the parameters of the model based on examples
from the user
The user is expected to label the patterns
The variety of activities is quite restricted
8
Proposed Technique
RFID (Radio Frequency Identification)
Cheap: Postage-stamp sized, forty-cent
Wireless and battery free
Activity modeling
Define an activity in terms of the probability and sequence
of the objects
Generate the models by translating textual definitions
Structured like recipes
Produced automatically by mining appropriate web sites
Mining models is part of a larger activity recognition system,
PROACT (Proactive Activity Toolkit)
9
Proposed Technique
Usage Model
Assumes that interesting objects in the environment
contain RFID tags (tens ~ hundreds)
Making a database entry mapping the tag ID to a name
Within a few years, many household objects may be RFID-
tagged before purchase, thus eliminating the overhead of
tagging
Medium-range readers (Tag-detecting Gloves) and
Long-range readers (Run robots, Carts, …)
PROACT uses the sequence and timing of object to
deduce what activity is happening
Likelihood of various activities, details of those activities,
degree of certainty, etc…
10
Proposed Technique
System Overview
PROACT provides an activity viewer for debugging
Real-time view of activities in progress
The sensor data seen
Changing of belief in each activity with the data
Inference Engine converts the activity models produced
by the mining engine into Dynamic Bayesian Networks
D. Patterson, L. Liao, D. Fox, H. Kautz, “Inferring High-Level
Behavior from Low-Level Sensors,” Ubicomp 2003.
11
Proposed Technique
Sensors and Models
Sensors
Use two different kinds of RFID readers
Long-range reader (mobile robot): map the location of objects
Short-range reader (glove): determine the objects that are
touched
Models
Each model (activity) is composed of a sequence (step) s1
~sn
Each step si has optional duration ti and object oij involved
along with the probability pij
12
Proposed Technique
The Model Extractor
Builds formal models of activities using directions
Directions are written in natural language by human
How-to (ehow.com), recipes (epicurious.com), training
manuals, protocols, etc.
Syntactic structure of directions
1. A title t for the activity
2. A textual list r1~rm, Each step ri has:
Possibly a special keyword delimiting duration di
What to do during the step: subset of the objects and duration
13
Proposed Technique
Converting Directions to Activity Models
Key steps
1. Labeling
Set label of the mined model to title of the directions
2. Parsing steps
Duration: Gaussian with mean = d, stdev = S(d, i, l )
Object Oi and Probability P
3. Tagged object filtering For example,
[“making tea”] has 24,200 matches, and
[“making tea” cup] has 7,340 matches, then
Functions conditional probability of a cup being involved in
making tea is 7340/2400 = 0.3
Object
Object extraction: WordNet ontology
Noun-phrase extraction: QTag tagger
Probability
Fixed probabilities
Google conditional probabilities (GCP)
14
Proposed Technique
Example
15
Evaluation
Mined models
ehow.com: 2300 directions
ffts.com: 400 recipes
epicurious.com: 18,600 recipes
Three strategies to approximate comprehensive
evaluation
Human activity-trace recognition
Activities of Daily Living (ADLs)
Inter-corpus consistency
Making cookies recipes
Intra-corpus distinguish-ability
Distinguish-ability within activity domains
16
Distinguish-ability
17
Evaluation
Human and inter-corpus trace recognition
ADLs domain
Many objects were not tagged, missed, and interleaved
Models were not perfect
Cookie domain
The identical recipe can have quite different structure
For some of the recipes, there is no counterpart in the other
corpus
18
Evaluation
Impact of techniques on accuracy
ADLs
Domain is fairly sparse, with many activities involving only
few object
Cookie domain
Each activity model involves many more objects
19
Evaluation
Impact of techniques on compactness
20
Summary and Future Work
An introduction to the idea of mining activity detection
from the web
Future work
Perform a more comprehensive evaluation
Improving the effectiveness of mined models
Include location
Synonymous words
Synsets (collections of synonymous words) can be extracted
from WordNet
21