Document Sample
ilp Powered By Docstoc
					       General Ideas in
Inductive Logic Programming

                   FOPI-RG - 19/4/05
ILP Problem settings
Bottom-up and top-down learning
Major Problem Areas
   Search Space
   Positive Only
Additional Points
Final Thoughts
  Inductive Logic Programming
• a set of examples, E
• background knowledge, BK
Produce a set of relations (clauses) using BK that
describe E.

ILP has two problem settings:
• nonmonotonic
• normal
        ILP Settings - nonmonotonic
    Given: Two sets, I+ and I-, of Herbrand
     interpretations (positive and negative

    Find: A theory which is
              true under each I in I+, and
              false under each I in I-.

Often only use positive examples for this problem setting
         ILP Settings - non-monotonic
     begin(e1)        begin(e2)      begin(e3)       begin(e4)
      penguin(e1)      carp(e2)       ostrich(e3)     eagle(e4)
      bird(e1)         fish(e2)       bird(e3)        bird(e4)
      feathers(e1)     scales(e2)     feathers(e3)    feathers(e4)
      swims(e1)        swims(e2)      runs(e3)        flies(e4)
     end(e1)          end(e2)        end(e3)         end(e4)

 Background knowledge:
(e.g. Claudien - complicated declarative language bias (Dlab) definition)
    bird(X):- feathers(X).
    swims(X):- fish(X); penguin(X).
             ILP Settings - normal
   Given: background knowledge, BK, and
          evidence E = E+ E

   Find: A hypothesis H such that
              BK       H     E+
               BK      H     E

BK can be:
• Extensional (ground facts)
• Intensional (more general predicate definitions)
• Something specialised (depending on the ILP engine)
              ILP Settings - normal
Examples:    Positive: bird(penguin)        Negative: bird(carp)
                       bird(eagle)                    bird(bat)
                       bird(crow)                     bird(horse)

Background knowledge:
lays_eggs(penguin). flies(eagle).   swims(carp).      runs(horse).
lays_eggs(crow). flies(crow).       swims(penguin). runs(ostrich).
lays_eggs(eagle). flies(bat).
lays_eggs(ostrich). fish(X) :- has_scales(X), swims(X).
lays_eggs(carp).    mammal(X):- warm_blooded(X), live_young(X).

Theory (one or more clauses): bird(penguin).
                              bird(X):- lays_eggs(X), flies(X).
                              bird(X):- lays_eggs(X), runs(X).
     Bottom-up vs Top-down
                 bird(X):- true.                       E

                bird(X):- flies(X).                 that flies
  bird(X):- lays_eggs(X).
   bird(X):- flies(X),…..lays_eggs(X).             that is v. like
                                                   an eagle

bird(penguin). bird(eagle). …..      bird(crow).     e+ (1 bird)

                false.                                 {}
               Bottom-Up Approach
   relative least general generalisation (rlgg)
               bird(X):- lays_eggs(X), flies(X), has(X, feathers),
                         has(X, beak), has(X, talons), makes_nest(X),

     bird(eagle)     bird(crow)

            bird(ostrich)                    bird(X):- lays_eggs(X),
                                     has(X, feathers), has(X, beak),
                                     has(X, talons), makes_nest(X),
                                     eats(X,Y), validate_food(X,Y).

Used in GOLEM [Muggleton, 90]
            Top-down Approach
Some ILP engines use standard top-down search algorithms:
depth-first, breadth-first, A*, etc.
  bird(X):- lays_eggs(X).
  bird(X):- flies(X).
  bird(X):- lays_eggs(X), flies(X).
We can improve efficiency by:
• setting a depth-bound (max clauselength).
• paying attention to clause evaluation scores - coverage, MDL.
    — re-ordering candidate clauses based on score
    — pruning candidate clauses below a score threshold
• etc.
               Problem areas
Most commonly encountered:

• Exploring large search spaces

• Positive-only data sets

• Noisy data
                 Search Space
The hypothesis space is bounded by:
   – Maximum clause length
   – Size of background knowledge (BK)

Techniques to reduce background knowledge include:
• Excluding redundant predicates
   – Feature subset selection
   – Inverse entailment
• Replacing existing BK with compound predicates
  (feature construction).
        Progol and Aleph’s Approach
    Uses inverse entailment.
       1. Randomly pick a positive example, p.
       2. Define the space of possible clauses that could
          entail that example.
            — Generate the bottom clause,
            — contains all the literals defined in BK that could
              cover p.
       3. Search this space.

(Progol and Aleph were originally C-Progol and P-Progol, I’m unclear on the
precise development history)
        Positive-only Learning
• Assume output completeness
  – Every other pattern in the example space is negative
• Bottom-up learning
• Clustering
  – e.g. Relational Distance Based Clustering

                                       ++ +
                         +        ++    ++
                        + +
          ++++ ++        +
        Positive-only Learning
• Assume output completeness
  – Every other pattern in the example space is negative
• Bottom-up learning
• Clustering
  – e.g. Relational Distance Based Clustering

                                       ++ +
                         +        ++    ++
                        + +
          ++++ ++        +
       Positive-only Learning
• Create artificial negative examples
  – Using constraints to represent artificial negatives
  – Use a relational distance measure

• Use Bayes theorem
  – Progol creates an SLP from the data set and uses
    this create random artificial negatives
                Noisy Data
• Techniques to avoid over-fitting.
  – Pre-pruning: limit length of clauses learned
  – Post-pruning: generalise/merge clauses that
    have a small cover set.
  – Leniency: don’t insist on a perfect theory

• Embed the uncertainty into the learning
  – Stochastic Logic Programs
  – Fuzzy ILP
             Numerical Reasoning
   e.g. bird(X):- number_of_legs(X,Y), lessthan(Y, 3).

A lot of ILP engines don’t handle numerical reasoning
without help.

• Lazy evaluation                  [Srinivasan & Camacho, 99]

• Farm it out to another process       [Anthony & Frisch, 97]

• (if possible) add predicates to the background knowledge
• First-Order Regression            [Karolic & Bratko, 97]
          Inventing Predicates
Some ILP engines can invent new predicates and add
  them to the existing BK.

e.g. Progol uses constraints to call a predicate
invention routine.

:- constraint(invent/2)?
invent(P,X):-   {complicated code that includes asserts}.

FOIL only uses extensional BK and so can’t use this
• Natural Language Processing
   – Part of speech tagging
   – Semantic parsing
   – Learning Language in Logic workshops
• Bioinformatics
   – Predicting toxicology (carcinogens)
   – Discovering rules governing the 3D topology of protein
• It doesn’t work very well for User Modelling.
              Final Thoughts
• There are lots of different approaches to ILP.
• Well known ILP systems:
  –   Progol (and Aleph)
  –   TILDE
  –   Claudien
  –   WARMR
  –   FOIL
  –   Merlin
                Further Reading
ILP Theory and Methods
Inverse Entailment and Progol
Analogical Prediction? – probably the other paper instead.
Stochastic Logic Programs
Lazy Evaluation
Interactive Theory Revision (Luc’s book)? Beginning chapters
Foundations of ILP, Nienhuys-Cheng and de Wolf
Claudien and Dlab
Learnability – pick a paper by Dzeroski

Shared By: