Lecture 09 - 3 Oct 2013 Huffman Coding Binary Encodings Symbols --> Binary strings Ascii: fixed length encoding "a" = 01100001 - fixed length code But: symbols are not distributed evenly. Variable length coding: give shorter codes to more frequent symbols. "a" 8% 0 "e" 12.7% 100 "j" 0.15% 101 "m" 2.4% 11 If end of character encoding isn't indicated by length, how can characters be separated? Prefix-free encoding: no character is a prefix of another character. Decoding table can be represented by a tree SHOW TREE How to find an optimal coding tree, given frequencies of symbols f1, f2, ...., fn ? cost of tree = SUM_{t=1 to n} f_t (depth of t-th symbol) Another formulation: sum over INTERNAL nodes (except root) as well: frequency of an internal node is the sum of frequency of its children: cost of tree = SUM_{n in Nodes\root} f*_n ADD INTERNAL FREQUENCIES TO TREE CLAIM: tree is FULL: every node is a leaf or has 2 children. WHY? CLAIM: the two nodes with the lowest frequency must be deepest in the tree == children of the deepest internal node. Why? Otherwise can rearrange tree to make it cheaper. SHOW EXAMPLE Claim: this means we can build the tree greedily from the bottom up, starting with the least frequent symbols. function Huffman(f) // input: // frequencies f[1..n] of symbols number 1 to n // output: Huffman coding tree // T[2n-1] is the root of the tree // for i in [1,n], T[i] corresponds to symbol i T = array[1..2n-1] of {value, left, right} H = makeHeap() for i = 1 to n: H.insert(i, f[i]) T[i].value = f[i] // Create leaf nodes for symbols T[i].left = 0 // null T[i].right = 0 // null for k = n+1 to 2n -1: // numbers for internal nodes i = H.deletemin() j = H.deletemin() // create node k with children i,j f[k] = f[i] + f[j] T[k].left = i T[k].right = j H.insert(k, f[k]) return T Measure of randomness: ENTROPY E = sum p_i log(1/p_i) Entropy = 0 means completely not random == completely compressible Entropy = 1 means completely random == not compressible Random coin: 1/2log2 + 1/2log2 = 1 -- cannot be compressed Always heads: 1 log 1 + 0 log INF = 0 + 0 = 0 easy to compress!