Lecture 09 - 3 Oct 2013
Huffman Coding
Binary Encodings
Symbols --> Binary strings
Ascii: fixed length encoding
"a" = 01100001 - fixed length code
But: symbols are not distributed evenly.
Variable length coding: give shorter codes to more frequent symbols.
"a" 8% 0
"e" 12.7% 100
"j" 0.15% 101
"m" 2.4% 11
If end of character encoding isn't indicated by length, how can characters be separated?
Prefix-free encoding: no character is a prefix of another character.
Decoding table can be represented by a tree
SHOW TREE
How to find an optimal coding tree, given frequencies of symbols
f1, f2, ...., fn ?
cost of tree = SUM_{t=1 to n} f_t (depth of t-th symbol)
Another formulation: sum over INTERNAL nodes (except root) as well:
frequency of an internal node is the sum of frequency of its children:
cost of tree = SUM_{n in Nodes\root} f*_n
ADD INTERNAL FREQUENCIES TO TREE
CLAIM: tree is FULL: every node is a leaf or has 2 children.
WHY?
CLAIM: the two nodes with the lowest frequency must be deepest in the tree
== children of the deepest internal node.
Why? Otherwise can rearrange tree to make it cheaper.
SHOW EXAMPLE
Claim: this means we can build the tree greedily from the bottom up, starting
with the least frequent symbols.
function Huffman(f)
// input:
// frequencies f[1..n] of symbols number 1 to n
// output: Huffman coding tree
// T[2n-1] is the root of the tree
// for i in [1,n], T[i] corresponds to symbol i
T = array[1..2n-1] of {value, left, right}
H = makeHeap()
for i = 1 to n:
H.insert(i, f[i])
T[i].value = f[i]
// Create leaf nodes for symbols
T[i].left = 0 // null
T[i].right = 0 // null
for k = n+1 to 2n -1: // numbers for internal nodes
i = H.deletemin()
j = H.deletemin()
// create node k with children i,j
f[k] = f[i] + f[j]
T[k].left = i
T[k].right = j
H.insert(k, f[k])
return T
Measure of randomness: ENTROPY
E = sum p_i log(1/p_i)
Entropy = 0 means completely not random == completely compressible
Entropy = 1 means completely random == not compressible
Random coin: 1/2log2 + 1/2log2 = 1 -- cannot be compressed
Always heads: 1 log 1 + 0 log INF = 0 + 0 = 0
easy to compress!