# ilp

Document Sample

```					       General Ideas in
Inductive Logic Programming

FOPI-RG - 19/4/05
Outline
ILP Problem settings
Bottom-up and top-down learning
Major Problem Areas
Search Space
Positive Only
Noise
Applications
Final Thoughts
Inductive Logic Programming
Given:
• a set of examples, E
• background knowledge, BK
Produce a set of relations (clauses) using BK that
describe E.

ILP has two problem settings:
• nonmonotonic
• normal
ILP Settings - nonmonotonic
Given: Two sets, I+ and I-, of Herbrand
interpretations (positive and negative
examples).

Find: A theory which is
true under each I in I+, and
false under each I in I-.

Often only use positive examples for this problem setting
ILP Settings - non-monotonic
Examples:
begin(e1)        begin(e2)      begin(e3)       begin(e4)
penguin(e1)      carp(e2)       ostrich(e3)     eagle(e4)
bird(e1)         fish(e2)       bird(e3)        bird(e4)
feathers(e1)     scales(e2)     feathers(e3)    feathers(e4)
end(e1)          end(e2)        end(e3)         end(e4)

Background knowledge:
(e.g. Claudien - complicated declarative language bias (Dlab) definition)
Relations:
bird(X):- feathers(X).
ILP Settings - normal
Given: background knowledge, BK, and
evidence E = E+ E

Find: A hypothesis H such that
BK       H     E+
BK      H     E

BK can be:
• Extensional (ground facts)
• Intensional (more general predicate definitions)
• Something specialised (depending on the ILP engine)
ILP Settings - normal
Examples:    Positive: bird(penguin)        Negative: bird(carp)
bird(eagle)                    bird(bat)
bird(crow)                     bird(horse)
bird(ostrich)

Background knowledge:
lays_eggs(eagle). flies(bat).
lays_eggs(carp).    mammal(X):- warm_blooded(X), live_young(X).

Theory (one or more clauses): bird(penguin).
bird(X):- lays_eggs(X), flies(X).
bird(X):- lays_eggs(X), runs(X).
Bottom-up vs Top-down
coverset
bird(X):- true.                       E

everything
bird(X):- flies(X).                 that flies
bird(X):- lays_eggs(X).
etc.
everything
bird(X):- flies(X),…..lays_eggs(X).             that is v. like
an eagle

bird(penguin). bird(eagle). …..      bird(crow).     e+ (1 bird)

false.                                 {}
Bottom-Up Approach
 relative least general generalisation (rlgg)
bird(X):- lays_eggs(X), flies(X), has(X, feathers),
has(X, beak), has(X, talons), makes_nest(X),
eats(X,rodents).

bird(eagle)     bird(crow)

bird(ostrich)                    bird(X):- lays_eggs(X),
has(X, feathers), has(X, beak),
has(X, talons), makes_nest(X),
eats(X,Y), validate_food(X,Y).

Used in GOLEM [Muggleton, 90]
Top-down Approach
Some ILP engines use standard top-down search algorithms:
bird(X):-.
bird(X):- lays_eggs(X).
bird(X):- flies(X).
bird(X):- lays_eggs(X), flies(X).
…
We can improve efficiency by:
• setting a depth-bound (max clauselength).
• paying attention to clause evaluation scores - coverage, MDL.
— re-ordering candidate clauses based on score
— pruning candidate clauses below a score threshold
• etc.
Problem areas
Most commonly encountered:

• Exploring large search spaces

• Positive-only data sets

• Noisy data
Search Space
The hypothesis space is bounded by:
– Maximum clause length
– Size of background knowledge (BK)

Techniques to reduce background knowledge include:
• Excluding redundant predicates
– Feature subset selection
– Inverse entailment
• Replacing existing BK with compound predicates
(feature construction).
Progol and Aleph’s Approach
Uses inverse entailment.
1. Randomly pick a positive example, p.
2. Define the space of possible clauses that could
entail that example.
— Generate the bottom clause,
— contains all the literals defined in BK that could
cover p.
3. Search this space.

(Progol and Aleph were originally C-Progol and P-Progol, I’m unclear on the
precise development history)
Positive-only Learning
• Assume output completeness
– Every other pattern in the example space is negative
• Bottom-up learning
• Clustering
– e.g. Relational Distance Based Clustering

++ +
+        ++    ++
+ +
++++ ++        +
+
++
Positive-only Learning
• Assume output completeness
– Every other pattern in the example space is negative
• Bottom-up learning
• Clustering
– e.g. Relational Distance Based Clustering

++ +
+        ++    ++
+ +
++++ ++        +
+
++
Positive-only Learning
• Create artificial negative examples
– Using constraints to represent artificial negatives
– Use a relational distance measure

• Use Bayes theorem
– Progol creates an SLP from the data set and uses
this create random artificial negatives
Noisy Data
• Techniques to avoid over-fitting.
– Pre-pruning: limit length of clauses learned
– Post-pruning: generalise/merge clauses that
have a small cover set.
– Leniency: don’t insist on a perfect theory

• Embed the uncertainty into the learning
mechanism
– Stochastic Logic Programs
– Fuzzy ILP
Numerical Reasoning
e.g. bird(X):- number_of_legs(X,Y), lessthan(Y, 3).

A lot of ILP engines don’t handle numerical reasoning
without help.

• Lazy evaluation                  [Srinivasan & Camacho, 99]

• Farm it out to another process       [Anthony & Frisch, 97]

• (if possible) add predicates to the background knowledge
• First-Order Regression            [Karolic & Bratko, 97]
Inventing Predicates
Some ILP engines can invent new predicates and add
them to the existing BK.

e.g. Progol uses constraints to call a predicate
invention routine.

:- constraint(invent/2)?
invent(P,X):-   {complicated code that includes asserts}.

FOIL only uses extensional BK and so can’t use this
method.
Applications
• Natural Language Processing
– Part of speech tagging
– Semantic parsing
– Learning Language in Logic workshops
• Bioinformatics
– Predicting toxicology (carcinogens)
– Discovering rules governing the 3D topology of protein
structures
• It doesn’t work very well for User Modelling.
Final Thoughts
• There are lots of different approaches to ILP.
• Well known ILP systems:
–   Progol (and Aleph)
–   TILDE
–   Claudien
–   WARMR
–   FOIL
–   Merlin
ILP Theory and Methods
Inverse Entailment and Progol
Analogical Prediction? – probably the other paper instead.
FOIL
TILDE
Stochastic Logic Programs
Lazy Evaluation
Interactive Theory Revision (Luc’s book)? Beginning chapters
Foundations of ILP, Nienhuys-Cheng and de Wolf
Claudien and Dlab
Learnability – pick a paper by Dzeroski

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 6 posted: 10/15/2012 language: English pages: 22