# Kolmogorov Complexity

Document Sample

```					Kolmogorov Complexity

Presented by: Wei Peng
Introduction
Source: Alan Turing in 1936 of the universal Turing machine

Kolmogorov complexity can be measured by the length of shortest
program for a universal Turing machine that correctly reproduces the
observed data S
KT(S) = |dT(S)| = |<T, w>|
Relationship between two streams

Kolmogorov
– Initiated by Kolmogorov with important later developments by Martin-Lof
and Chaitin. The body of information is usually assumed to be a finite
string of binary digits S.
– Motivation: information theory

Solomonoff
– Chronologically the first, springs from the work of Solomonoff. The intent
is not to measure the complexity of S, but rather to develop a probability
distribution over the set of finite binary strings.
– Motivation: inductive inference and artificial intelligence
Relationship between two streams
Output tape: one-way, written, no over-written
Input and output tapes: binary alphabet with no blank or delimiter symbol in

Stream one: The turing machine is required to stop
KT(S) = |dT(S)|
Stream two: Its further action is unspecified
PT(S) =   
I
(2  # I )
Definition of Information
A = 0101010101010101010101010101
B = 1110010110100011101010000111

The length of the description is the combined length of
representing M and w
Definition 6.20
Let x be a binary string. The minimal description of x, written d(x), is the
shortest string <M, w> where TM M on input w halts with x on its tape. If
several such strings exist, select the lexicographically first among them.
The descriptive complexity (Kolmogorov complexity) of x, written K(x), is
K(x) = |d(x)|
Some proofs of Kolmogorov complexity

cx[ K ( x) | x | c]
Proof: M is a Turing machine that halts as soon as it is started. This
machine
computes the identity function – its output is the same as its input. A
description of x is simply <M>x. Letting c be the length of <M>
completes the proof.

cx[ K ( xx)  K ( x)  c]
Proof: M=“On input <N, w> where N is a TM and w is a string:
1. Run N on w until it halts and produces an output string s.
2. Output the string ss.”
K(xx)= |<M>| + |d(x)| = c + K(x)
Some proofs of Kolmogorov complexity

cx, y[ K ( xy)  2K ( x)  K ( y)  c]
Proof: We construct a TM M that breaks its input w into two separate
descriptions. The bits of the first description d(x) are all doubled and
terminated with string 01 before the second description d(y) appears.
Once both descriptions are obtained, they are run to obtain the string
X and y and the output xy is produced.

2log(K(x)) + K(x) + K(y) +c

x[ K ( x )  K p ( x )  c ]
Incompressible strings and randomness

Let x be a string. Say that x is c-compressible if K ( x) | x | c
If x is not c-compressible, say that x is incompressible by c. If x is
incompressible by 1, say that x is incompressible.

• Incompressible strings of every length exist.
n
Proof: The number of binary strings of length n is 2
Each description is a nonempty binary string, so the number of descriptions of
length less than n is at most the sum of the number of strings of each length
up to n-1, or

2
0  i  n 1
i
 1  2  4...  2 n 1  2 n  1
Incompressible strings and randomness
For some constant b, for every string x, the minimal description d(x) of x is incompressible by
b.
Proof: consider the following TM M:
M = “On input <R, y>, where R is a TM and y is a string:
1. Run R on y and reject if its output is not of the form <S, z>.
2. Run S on z and halt with its output on the tape.”
Let b be |<M>|+1. We show that b satisfies the theorem. Suppose to the contrary that d(x) is b-
compressible for some string x. Then

| d (d ( x)) || d ( x) | b
But then <M>d(d(x)) is a description of x whose length is at most
| M |  | d (d ( x)) | (b  1)  (| d ( x) | b) | d ( x) | 1
This description of x is shorter that d(x), contradicting the latter’s minimality.
Randomness
Random

Complexity theory: A string S is random for a UTM T if KT(S) ≥ |S|-c, where c is a
small constant chosen to impose a „significance‟ requirement on the concept of
non-randomness

Work on applying the algorithmic theory of randomness to practical problems of
computer learning is under way in the Computer Learning Research Center at
Royal Holloway, University of London. The detail information can be accessed at
http://www.clrc.rhbnc.ac.uk

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 7 posted: 9/20/2011 language: English pages: 10