# Handwritten notes from class 9 � Monday 24 by Q4u7m1h

VIEWS: 25 PAGES: 4

• pg 1
```									        Handwritten notes from class 9 – Monday 24.09.01
(typed in by Hans Arvid Johansen)

Statistics:

●   3 people and 3 cats have been observed leaving the room

●   Statistics from past observations would have led us to believe there was no
chance of seeing a llama leave the room

●   Events are independent if the occurrence of one has no influence on the
occurrence of the other

●   Example: Tossing 2 fair coins

Probability of heads = ½ (one coin)
Probability of any outcome:

H+H     =½*½=¼
H+T     =½*½=¼
T+H     =½*½=¼
T+T     =½*½=¼

Entropy:
●   Entropy (H) of the probability distribution pi measures the amount of
uncertainty, surprise, in information that a distribution of outcomes contains

z
1
H=     pi log
i 1            pi
pi is probability of a symbol occurring,

z is number of symbols in the symbol set.
Shannon’s Entropy Law

1
Entropy Encoding Example

The entropy of a binary event
with two outcomes as a
function of the probability of
one outcome

H  empty

Probability of an event
M=1 F=0
←       111010011011       1 bit per patron

Example: Men’s clothing store

A study of 1000 customers reveals that
800 patrons where male and that
200 patrons where female

Here we see H = 0.72 bits of information

●     But you need a whole bit to represent the next customer that comes in
the door

1 = Male     0 = Female  (0.8) 1 + (0.2) 1 = 1 bit / patron

●         But if you group results you need less data bits to store the information
●         So look at the probability of the next to customers that enter:

A   M-M = (0.8 * 0.8) = 0.64               0
B   M-F = (0.8 * 0.2) = 0.16               10         Assigning an unique
C   F-M = (0.2 * 0.8) = 0.16               110        code, decodable
D   F-F = (0.2 * 0.2) = 0.04               111

← 0 10 10 110 100 is events A B B C B A
What is the compression?
(0.64) 1bit + (0.16) 2bits + (0.16) 3bits + (0.04) 3bits = 1.56 bits/events

 1.56 / 2 = 0.78 bits/patron        The best you can compress is 0.72

2
Entropy encoding:      - depends on statistics
- knowing the probability of events from samples

Universal coding:      - assumes any stream of data in a given document is a typical
stream for that document
- good compression is achieved with very large documents

Lempel – Ziv:          - is an example of universal coding
- we code series of “prefix string” plus one new bit

Data string: 101011011010101011
Parse string: 1,0,10,11,01,101,010,1011         unique decodable strings

Prefix   Code
Null     000        The eight bit phrases can be described by pairs of
1        001        bit groups of 3-bit prefix phrase + 1 additional bit
0        010        to complete the phrase
10       011
11       100
01       101
101      110
010      111

(000,1)(000,0)(001,0)(001,1)(010,1)(011,1)(101,0)(110,1)

● In this case there was no compression
But with larger initial strings, we get more savings as we move along, because the
prefixes become very long and they are represented as small numerical codes

● Unix OS “compress” uses this

3
For each rose,└┘ a rose is a rose

Vocabulary                      Occurrences

a                                   2
each                                1
is                                  1
rose                                3
,└┘                                 1
for                                 1

Canonical code:                     - Left is longer than right
(better)                            - Increasing probability from left to right
- Impose order on the branches
(because there are many possible trees)

0

0
1
rose         prob. 0.333
0               1
a                prob. 0.222

0               1

1 0 1
each ,└┘ for is                            prob. 0.111

Code: Word:

0000           each                        Codes of the same length
0001           ,└┘                         ( = better) are in binary order
0010           for
0011           is                          Read ’00’ you know it’s 4 bits
01             a                           long. ’00’ can index the 4 bit
1              rose                        codes

Efficient for encoding and
decoding – uses less memory

4

```
To top