Satoshi Yoshida* and Takuya Kida* *Hokkaido University Graduate school of Information Science and Technology Division of Computer Science

Transcription

1 Stoshi Yoshid* nd Tkuy Kid* *Hokkido University Grdute school of Informtion Science nd Technology Division of Computer Science 1

2 Compressed Dt Serch Directly Progrm Serching on Compressed Dt Compressed Text Fixed Length Huffmn Code Vrible Length Fixed Length FF Code (Fixed length to Fixed length code) FV Code (Fixed length to Vrible length code) Input Text Vrible Length Tunstll Code VF Code (Vrible length to Fixed length code) VV Code (Vrible length to Vrible length code) 2

3 AIVF code using multiple prse trees [Ymmoto nd Yokoo, 2001] YY code Improves compression rtio considerbly utilizing context between blocks AIVF code [Ymmoto nd Yokoo, 2001] utilizing unused codewords Tunstll code [Tunstll, 1967] 3

4 multiple prse trees huge time nd memory reltively to the number of kind of chrcters (k) in the input text VMA tree k-1 prse trees proposing method 4

5 We cn reduce the totl number of nodes in comprison to YY code by using VMA tree. We cn reduce the totl number of nodes nd compression time considerbly lso in experiments. We found n upper bound nd lower bound of the number of nodes in VMA tree. 5

6 Let Σ be finite lphbet. Elements in lphbet re sorted in descending order of their probbilities. We ssume informtion source is memoryless. We simply sy encoding encoding to the sequence on {0, 1}. bbbbc Σ={, b, c} 6

7 input text: bbbbc Ech brnch is lbeled by symbol in Σ={, b, c}. 001 b c Ech lef node nd incomplete internl node hs codeword. b 100 c 110 Incomplete internl node An internl node which doesn t hvekchildren. 111 Input text is prsed into blocks nd we output codeword corresponding to the block. compressed dt sequence:

8 We switch multiple prse trees ccording to the context. b c b c b c b c b c

9 Problem: time nd spce consuming We construct k 1 prse trees. We cn reduce totl number of nodes by shring nodes. We need to mrk ech node n in VMA tree in order to tell which trees the node belongs to. We cn relize tht by holding the lest i such tht n belongs to T i. T 0 T 1 T 2 T k 2 multiple prse trees VMA tree 9

10 From this theorem, we cn tell which trees node belongs to esily. Theorem (i) Let S be the subtree of T j i tht consists of ll the nodes under the node corresponding with j, which is ( i+1) (i) direct child of the root. Then S i + j completely covers S i + j. ( i) ( i+ 1) We denote this reltion by S ps. i+ j i+ j T i T i +1 ( i) S i + 1 ( i) S i + 2 ( i+ 1) S i + 2 ( i) ( i) Sk S (i) 2 k 1 ( i+ 1) ( i+ 1) S ( i+1) 2 k 1 10

11 S 1 We cll the integrted prse tree VMA tree. T0 T T 1 k S ( k 2) 1 From the theorem, we hve: ( k 2) S S S S S S k 2 k 1 k = S ps ps plps plps 1, 2 3 M plps T V = S 2 ps, (2) 3 ( k 3) k 2 ( k 2) k 1 ( k 2) k = S 3 = S = S = S k, k 2 k 1.,, multiple prse trees S1 2 1 VMA tree 11

12 Theorem The number of reduced nodes is not less thn. 12

13 T 1 T k 3 T k 2 S 1 S 2 T S 2 S ( k 3) 2 ( k 3) 1 k 1 k 2 2 Summtion: ( k 3) T V ( k 2) 1 ( k 2) Ech reduced subtree hs t lest one node. S 1 S 2 ( k 3) 2 ( k 2) 1 ( k 2) 13

14 Theorem The number of nodes in VMA tree is 1 1 not less thn. 14

15 Lrgest subtree nd it remins in VMA tree. The VMA tree is smllest when Pr Pr Pr. 15

16 T V #of codewords: S 1 S 2 1 ( k 3) 2 3 ( k 2) 1 2 ( k 2) 2 #of nodes: Summtion is not less thn. 16

17 Comprison Totl number of nodes (Exp1) Compression times (Exp2) Totl number of nodes nd upper bound nd lower bound of nodes in VMA tree on rndom sequence (Exp3) Algorithms YY coding (YY) Encoding using VMA tree (VMA) Environments CPU: Intel Pentium 4 Processor 3.0GHz Hyper Threding Memory: 2GB OS: Debin GNU/Linux 5.0 Lnguge: C++ Compiler: g++4.3 Codeword length: 12 bits 17

18 The Cnterbury Corpus file size (in bytes) k content lice29.txt 152, English text syoulik.txt 125, Shkespere cp.html 24, HTML source fields.c 11, C source grmmr.lsp 3, LISP source kennedy.xls 1,029, Excel Spredsheet lcet10.txt 424, Technicl writing plrbn12.txt 481, Poetry ptt5 513, CCITT test set sum 38, SPARC Executble xrgs.1 4, GNU mnul pge 18

19 VMA YY times times totl mount of nodes k :

20 times 3 times k : VMA YY 20 compression time (sec)

21 1200 the number of nodes on uniform distribution 1200 the number of nodes on Zipf distribution totl number of nodes totl number of nodes VMA 600 YY upper 400 bound lower bound 200 VMA YY upper bound lower bound lphbet size lphbet size 21

22 We showed tht we cn reduce the totl number of nodes in comprison to YY code by using VMA tree. We lso showed tht we cn reduce the totl number of nodes nd times considerbly in experiments. We found n upper bound nd lower bound of the number of nodes in VMA tree. Applying this ide to STVF coding [Kid, DCC2009] Finding tighter bounds 22