Kolmogorov Complexity

Document Sample
Kolmogorov Complexity Powered By Docstoc
					Kolmogorov Complexity

        Presented by: Wei Peng
Source: Alan Turing in 1936 of the universal Turing machine

Kolmogorov complexity can be measured by the length of shortest
program for a universal Turing machine that correctly reproduces the
observed data S
KT(S) = |dT(S)| = |<T, w>|
   Relationship between two streams

– Initiated by Kolmogorov with important later developments by Martin-Lof
  and Chaitin. The body of information is usually assumed to be a finite
  string of binary digits S.
– Motivation: information theory

– Chronologically the first, springs from the work of Solomonoff. The intent
  is not to measure the complexity of S, but rather to develop a probability
  distribution over the set of finite binary strings.
– Motivation: inductive inference and artificial intelligence
        Relationship between two streams
Input tape: one-way, read-only
Output tape: one-way, written, no over-written
Input and output tapes: binary alphabet with no blank or delimiter symbol in
   addition to zero and one

   Stream one: The turing machine is required to stop
               KT(S) = |dT(S)|
   Stream two: Its further action is unspecified
                 PT(S) =   
                              (2  # I )
                 Definition of Information
A = 0101010101010101010101010101
B = 1110010110100011101010000111

  The length of the description is the combined length of
  representing M and w
  Definition 6.20
    Let x be a binary string. The minimal description of x, written d(x), is the
     shortest string <M, w> where TM M on input w halts with x on its tape. If
     several such strings exist, select the lexicographically first among them.
     The descriptive complexity (Kolmogorov complexity) of x, written K(x), is
     K(x) = |d(x)|
 Some proofs of Kolmogorov complexity

  cx[ K ( x) | x | c]
Proof: M is a Turing machine that halts as soon as it is started. This
computes the identity function – its output is the same as its input. A
description of x is simply <M>x. Letting c be the length of <M>
completes the proof.

  cx[ K ( xx)  K ( x)  c]
Proof: M=“On input <N, w> where N is a TM and w is a string:
          1. Run N on w until it halts and produces an output string s.
          2. Output the string ss.”
K(xx)= |<M>| + |d(x)| = c + K(x)
     Some proofs of Kolmogorov complexity

cx, y[ K ( xy)  2K ( x)  K ( y)  c]
Proof: We construct a TM M that breaks its input w into two separate
descriptions. The bits of the first description d(x) are all doubled and
terminated with string 01 before the second description d(y) appears.
Once both descriptions are obtained, they are run to obtain the string
X and y and the output xy is produced.

2log(K(x)) + K(x) + K(y) +c

x[ K ( x )  K p ( x )  c ]
        Incompressible strings and randomness

   Let x be a string. Say that x is c-compressible if K ( x) | x | c
   If x is not c-compressible, say that x is incompressible by c. If x is
   incompressible by 1, say that x is incompressible.

• Incompressible strings of every length exist.
Proof: The number of binary strings of length n is 2
Each description is a nonempty binary string, so the number of descriptions of
length less than n is at most the sum of the number of strings of each length
up to n-1, or

 0  i  n 1
                     1  2  4...  2 n 1  2 n  1
      Incompressible strings and randomness
 For some constant b, for every string x, the minimal description d(x) of x is incompressible by
 Proof: consider the following TM M:
M = “On input <R, y>, where R is a TM and y is a string:
     1. Run R on y and reject if its output is not of the form <S, z>.
     2. Run S on z and halt with its output on the tape.”
Let b be |<M>|+1. We show that b satisfies the theorem. Suppose to the contrary that d(x) is b-
 compressible for some string x. Then

    | d (d ( x)) || d ( x) | b
 But then <M>d(d(x)) is a description of x whose length is at most
    | M |  | d (d ( x)) | (b  1)  (| d ( x) | b) | d ( x) | 1
 This description of x is shorter that d(x), contradicting the latter’s minimality.

Complexity theory: A string S is random for a UTM T if KT(S) ≥ |S|-c, where c is a
small constant chosen to impose a „significance‟ requirement on the concept of

Work on applying the algorithmic theory of randomness to practical problems of
computer learning is under way in the Computer Learning Research Center at
Royal Holloway, University of London. The detail information can be accessed at

Shared By: