HaLoop: Efficient Iterative Data Processing On Large Clusters

Transcription

1 HaLoop: Efficient Iteative Data Pocessing On Lage Clustes Yingyi Bu, UC Ivine Bill Howe, UW agda Balazinska, UW ichael D. Enst, UW Hoizon QuickTime ᆰ and a decompesso ae needed to see this pictue. ay-2011, UC Bekeley Cloud Computing Semina

2 Outline otivation Caching & scheduling Fault-toleance Pogamming model Related wok Conclusion Cloud Computing Pojects in UCI

3 otivation apreduce can t expess ecusion/iteation Lots of inteesting pogams need loops gaph algoithms clusteing machine leaning ecusive queies (CTEs, datalog, WITH clause) Dominant solution: Use a dive pogam outside of apreduce Hypothesis: making apreduce loop-awae affods optimization lays a foundation fo scalable implementations of ecusive languages 05/03/11 Yingyi Bu, UCI 3

4 Example 1: PageRank Rank Table R 0 ul ank Rank Table R 3 Linkage Table L ul_sc ul_dest R i+1 π(ul_dest, γ ul_dest SU(ank)) R i.ank = R i.ank/γ ul COUNT(ul_dest) ul ank R i.ul = L.ul_sc R i L 05/03/11 Yingyi Bu, UCI 4

5 PageRank Implementation on apreduce Join & compute ank R i Aggegate Fixpoint evaluation Count L-split0 L-split1 i=i+1 Conveged? Client done 05/03/11 Yingyi Bu, UCI 5

6 What s the poblem? R i m Count m L-split0 L-split1 1. m m L and Count ae loop invaiants, but 1. They ae loaded on each iteation 2. They ae shuffled on each iteation 3. Also, fixpoint evaluated as a sepaate apreduce job pe iteation 05/03/11 Yingyi Bu, UCI 6

7 Example 2: Tansitive Closue Fiend Find all tansitive fiends of Eic R 0 {Eic, Eic} (semi-naïve evaluation) R 1 R 2 R 3 {Eic, Elisa} {Eic, Tom Eic, Hay} {} 05/03/11 Yingyi Bu, UCI 7

8 Tansitive Closue on apreduce (compute next geneation of fiends) Join (emove the ones we ve aleady seen) Dup-elim S i Fiend0 Fiend1 Anything new? i=i+1 Client done 05/03/11 Yingyi Bu, UCI 8

9 What s the poblem? Join Dupe-elim S i Fiend0 Fiend Fiend is loop invaiant, but 1. Fiend is loaded on each iteation 2. Fiend is shuffled on each iteation 05/03/11 Yingyi Bu, UCI 9

10 Example 3: k-means k i = k centoids at iteation i P0 k i P1 k i P2 k i k i - k i+1 < theshold? i=i+1 Client done k i+1 05/03/11 Yingyi Bu, UCI 10

11 What s the poblem? P0 k i k i = k centoids at iteation i P1 k i P2 1. k i k i - k i+1 < theshold? i=i+1 Client done k i+1 P is loop invaiant, but 1. P is loaded on each iteation 05/03/11 Yingyi Bu, UCI 11

12 Push loops into apreduce! Achitectue Cache loop-invaiant data Scheduling Fault-toleance Pogamming odel 05/03/11 Yingyi Bu, UCI 12

13 HaLoop Achitectue 05/03/11 Yingyi Bu, UCI 13

14 Inte-iteation caching Reduce output cache (RO) appe output cache (O) appe input cache (I) Reduce input cache (RI) 05/03/11 Yingyi Bu, UCI 14

15 RI: Reduce Input Cache Povides: Access to loop invaiant data without map/shuffle Data: Reduce function Assumes: 1. Static patitioning (implies: no new nodes) 2. Deteministic mappe implementation PageRank Avoid loading and shuffling the web gaph at evey iteation Tansitive Closue Avoid loading and shuffling the fiends gaph at evey iteation K-means No help 05/03/11 Yingyi Bu, UCI 15

16 Reduce Input Cache Benefit Fiends-of-fiends quey Billion Tiples Dataset (120GB) 90 small instances on EC2 Oveall un time 05/03/11 Yingyi Bu, UCI 16

17 Reduce Input Cache Benefit Fiends-of-Fiends quey Billion Tiples Dataset (120GB) 90 small instances on EC2 Livejounal, 12GB Join step only 05/03/11 Yingyi Bu, UCI 17

18 Reduce Input Cache Benefit Fiends-of-fiends quey Billion Tiples Dataset (120GB) 90 small instances on EC2 Livejounal, 12GB Reduce and Shuffle of Join Step 05/03/11 Yingyi Bu, UCI 18

19 RO: Reduce Output Cache Povides: Distibuted access to output of pevious iteations Used By: Fixpoint evaluation Assumes: 1. Patitioning constant acoss iteations 2. Reduce output key functionally detemines Reduce input key PageRank Allows distibuted fixpoint evaluation Obviates exta apreduce job Tansitive Closue No help K-means No help 05/03/11 Yingyi Bu, UCI 19

20 Reduce Output Cache Benefit Fixpoint evaluation (s) Iteation # Iteation # Livejounal dataset 50 EC2 small instances Feebase dataset 90 EC2 small instances 05/03/11 Yingyi Bu, UCI 20

21 I: appe Input Cache Povides: Access to non-local mappe input on late iteations Data fo: ap function Assumes: 1. appe input does not change PageRank No help Tansitive Closue No help K-means Avoids non-local data eads on iteations > 0 05/03/11 Yingyi Bu, UCI 21

22 appe Input Cache Benefit 5% non-local data eads; ~5% impovement Howeve, Facebook has 70% non-local data eads!! 05/03/11 Yingyi Bu, UCI 22

23 Loop-awae Task Scheduling Input: Node node, int iteation Global vaiable: Hashap<Node, List<Paition>> last, Hashaph<Node, List<Patition>> cuent 1: if (iteation ==0) { 2: Patition pat = StandadapReduceSchedule(node); 3: cuent.add(node, pat); 4: }else{ 5: if (node.hasfullload()) { 6: Node substitution = findneabynode(node); 7: last.get(substitution).addall(last.emove(node)); 8: etun; 9: } 10: if (last.get(node).size()>0) { 11: Patition pat = last.get(node).get(0); 12: schedule(pat, node); 13: cuent.get(node).add(pat); 14: list.emove(pat); 15: } 16: } The same as apreduce Find a substitution Iteation-local Schedule 05/03/11 Yingyi Bu, UCI 23

24 Fault-toleance (task failues) Cache eloading Task failue Task e-execution 05/03/11 Yingyi Bu, UCI 24

25 Fault-toleance (node failues) node failue Cache eloading Task e-execution 05/03/11 Yingyi Bu, UCI 25

26 Pogamming odel appe/educe stay the same! Touch points Input/Output: fo each <iteation, step> Cache filte: which tuple to cache? Distance function: optional Nested job containing child jobs as loop body inimize exta pogamming effots 05/03/11 Yingyi Bu, UCI 26

27 Related Wok: Twiste [Ekanayake HPDC 2010] Pipelining mappe/educe Temination condition evaluated by main() 13. while(!complete){ 14. monito = dive.unapreducebcast(cdata); 15. monito.monitotillcompletion(); 16. DoubleVectoData newcdata = ((KeansCombine) dive.getcuentcombine()).getresults(); 17. totaleo = geteo(cdata, newcdata); 18. cdata = newcdata; 19. if (totaleo < THRESHOLD) { O(k) 20. complete = tue; 21. beak; 22. } 23. } 05/03/11 Yingyi Bu, UCI 27

28 In Detail: PageRank (Twiste) un R tem. cond. while (!complete) { // stat the pageank map educe pocess monito = dive.unapreducebcast(new BytesValue(tmpCompessedDvd.getBytes())); monito.monitotillcompletion(); // get the esult of pocess newcompesseddvd = ((PageRankCombine) dive.getcuentcombine()).getresults(); // decompess the compessed pageank values } newdvd = decompess(newcompesseddvd); tmpdvd = decompess(tmpcompesseddvd); totaleo = geteo(tmpdvd, newdvd); O(N) in the size of the gaph // get the diffeence between new and old pageank values if (totaleo < toleance) { complete = tue; } tmpcompesseddvd = newcompesseddvd; 05/03/11 Yingyi Bu, UCI 28

29 Related Wok: Spak [Zahaia HotCloud 2010] Reduction output collected at dive pogam does not cuently suppot a gouped educe opeation as in apreduce val spak = new SpakContext(<esos maste>) all output sent va count = spak.accumulato(0) to dive. fo (i <- spak.paallelize(1 to 10000, 10)) { val x = ath.andom * 2-1 val y = ath.andom * 2-1 if (x*x + y*y < 1) count += 1 } pintln("pi is oughly " + 4 * count.value / ) 05/03/11 Yingyi Bu, UCI 29

30 Related Wok: Pegel [alewicz SIGOD 2010] Gaphs only clusteing: k-means, canopy, DBScan Assumes each vetex has access to outgoing edges So an edge epesentation Edge(fom, to) equies offline pepocessing pehaps using apreduce 05/03/11 Yingyi Bu, UCI 30

31 Related Wok: BOO [Alvao EuoSys 10] Distibuted computing based on Ovelog (Datalog + tempoal logic + moe) Recusion suppoted natually app: API-compliant implementation of R 05/03/11 Yingyi Bu, UCI 31

32 Conclusions Relatively simple changes to apreduce/hadoop can suppot iteative/ecusive pogams TaskTacke (Cache management) Schedule (Cache awaeness) Pogamming model (multi-step loop bodies, cache contol) Optimizations Caching educe input ealizes the lagest gain Good to eliminate exta apreduce step fo temination checks appe input cache benefit inconclusive; need a busie cluste Futue Wok Iteation & Recusion on top of Hyacks coe! 05/03/11 Yingyi Bu, UCI 32

33 Hyacks [Boka et al., ICDE'11] Patitioned-Paallel Platfom fo data-intensive computing Flexible (DAGs, location constaints) Extensible ( mico-kenel coe, online-aggegation plugin (VLDB'11), B-tee plugin, R-tee plugin, Dataflow plugin...), an iteation/ecusion plugin? Jobs Dataflow DAG of opeatos and connectos Can set location constaints Can use a libay of opeatos: joins, goup-by, sot and so on V.s. competitos V.s. Hadoop: moe flexible model and less pessimistic V.s. Dyad: suppot data as fist class citizens 05/03/11 Yingyi Bu, UCI 33

34 Questions? 34