Markov Logic and
Deep Networks
Pedro Domingos
Dept. of Computer Science & Eng.
University of Washington
Markov Logic Networks
Basic idea: Use first-order logic to
compactly specify large non-i.i.d. models
MLN = Set of formulas with weights
Formula = Feature template (Vars→Objects)
E.g., HMM: State(+s,t) ^ State(+s',t+1)
Obs(+o,t) ^ State(+s,t)
1
P( x) exp wi ni ( x)
Z i
Weight of formula i No. of true instances of formula i in x
State of the Art in MLNs
Many algorithms for learning and inference
Inference: Millions of variables, billions of features
Learning: Generative, discriminative, max-margin, etc.
Best-performing solutions in many application areas
Natural language, robot mapping, social networks,
computational biology, activity recognition, etc.
Open-source software/Web site: Alchemy
alchemy.cs.washington.edu
Book: Domingos & Lowd, Markov Logic,
Morgan & Claypool, 2009.
Deep Uses of MLNs
Very large scale inference
Defining architecture of deep networks
Adding knowledge to deep networks
Transition to natural language input
Learning with many levels of hidden variables
MLNs for Deep Learning
Basic idea: Use small amounts of knowledge
and large amounts of joint inference
to make up for lack of supervision
Relational clustering:
Cluster objects with similar relations to similar objects
Cluster relations that hold between similar sets of objects
Coreference resolution: Outperforms supervised
approaches on MUC and ACE benchmarks
Semantic network extraction: Learns thousands of
concepts and relations from millions of Web triples
Example:
Unsupervised Semantic Parsing
Goal: Read text and answer questions
No supervision or annotation
Input: Dependency parses of sentences
(Nodes→Unary predicates / Edges→Binary preds.)
Outputs: Semantic parser and knowledge base
Basic idea: Cluster expressions with similar
subexpressions
Maps syntactic variants to common meaning
Discovers its own meaning representation
“Part of” lattice of clusters
Applied to corpus of biomedical abstracts
Three times more correct answers than next best