Caching and Witback Policis in Paalll Fil Systms David Kotz Cala Schlatt Ellis Dpt. of Math and Comput Scinc Dpt. of Comput Scinc Datmouth Collg Duk Univsity Hanov, NH 03755-3551 Duham, NC 27706 David.Kotz@Datmouth.du cala@cs.duk.du Abstact Impovmnts in th pocssing spd of multipocssos a outpacing impovmnts in th spd of disk hadwa. Paalll disk I/O subsystms hav bn poposd as on way to clos th gap btwn pocsso and disk spds. Such paalll disk systms qui paalll l systm softwa to avoid pfomanc-limiting bottlncks. W discuss cach managmnt tchniqus that can b usd in a paalll l systm implmntation. W xamin sval witback policis, and giv sults of xpimnts that tst thi pfomanc. 1 Intoduction As computs gow mo powful, it bcoms incasingly dicult to povid sucint I/O bandwidth to kp thm unning at full spd fo lag poblms, which may consum immns amounts of data. Disk I/O has always bn slow than pocssing spd, and cnt tnds hav shown that impovmnts in th spd of disk hadwa a not kping up with th incasing aw spd of pocssos. This widning accss-tim gap is known as th I/O cisis [13, 16]. Th poblm is compoundd in typical paalll achitctus that multiply th pocssing and mmoy capacity without balancing th I/O capabilitis. Th most pomising solution to th I/O cisis is to xtnd paalllism into th I/O subsystm. On such appoach is to connct many disks to th comput in paalll, spading individual ls acoss all disks. Paalll disks could povid a signicant boost in pfomanc possibly qual to th dg of paalllism, if th a no signicant bottlncks in th I/O subsystm and if th I/O qusts gnatd by applications can b mappd into low-lvl opations that div th availabl paalllism. Thus, th st challng to th dsigns of a multipocsso l systm is to congu paalll disk hadwa to avoid bottlncks (.g., shad busss), and to avoid futh bottlncks in th systm softwa. An ctiv l systm fo a multipocsso must itslf b fully paalll to scal with additional pocssos o disks. Th scond challng is to mak this xtnsiv disk hadwa This sach was suppotd in pat by NSF gants CCR- 8721781 and CCR-8821809 and DARPA/NASA subcontact of NCC2-560. bandwidth asily availabl to application pogams. To mt ths challngs w popos a highly paalll l systm implmntation that incopoats caching and pftching as a mans of dliving th bnts of a paalll I/O achitctu to th us pogams. This pap concntats on multipocsso l systms intndd fo scintic applications. Ths applications typically push th lading dg of computing tchnology, such as multipocssos, placing tmndous dmands on both CPU and I/O systms. Most l caching studis hav xamind gnalpupos wokloads (.g., [16]), wh ls a much small [12, 5]. Th paalll nvionmnt and wokload ais a numb of qustions: A cachs usful fo paalll scintic applications using paalll l systms? If so, in what way? What a th appopiat managmnt policis? Dint wokload chaactistics, including a nw fom of locality, lad us to nw policis. Th squntial accss pattns in th wokload suggst pftching and wit-bhind. Pftching is th focus of [9,, 8], and wit-bhind is th focus of this pap. What policis a most appopiat fo buing wits fo ths paalll scintic-application wokloads? Do witbhind and dlayd witback hlp? In what way? This pap xamins ths issus, dns som nw policis, and pots sults fom xpimnts with ths policis. In th nxt sction w povid mo backgound infomation on paalll I/O, caching, and l systm wokloads. In Sction 3 w dscib th tstbd, th wokload, th xpimntal mthods, and th cach managmnt policis. In Sction 4 w psnt th xpimnts, pfomanc masus, and sults. Sction 5 concluds. 2 Backgound Much of th pvious wok in I/O hadwa paalllism involvs disk stiping. In this tchniqu, data of a l a intlavd acoss numous disks and accssd synchonously in paalll [15, 7, 13]. Ths schms ly on a singl contoll to manag all of th disks. Fo multipocssos, on fom of paalll disk achitctu is basd on th notion of paalll, indpndnt disks, using multipl convntional disk dvics 1
addssd indpndntly and attachd to spaat pocssos. Th ls may b intlavd ov th disks, but th multipl contolls and indpndnt accss to th disks mak this tchniqu dint fom disk stiping. Exampls of this achitctu includ th Concunt Fil Systm [14, 6] fo th Intl ipsc/2 multipocsso, and th Bidg [4, 3] l systm fo th BBN Butty paalll comput. Fil caching is a tchniqu usd in most modn l systms. Caching has not bn studid fo paalll l systms, but Alan Smith has xtnsivly studid disk caching in unipocssos with gnal-pupos wokloads. In [16], his simulations show that disk caching is an ctiv way to boost th pfomanc (as masud by th cach miss atio) of th I/O subsystm (.g., an 8 MByt cach can svic 80{90% of I/O qusts). Fil accss pattns hav nv bn studid fo paalll computs, but hav bn studid xtnsivly fo unipocssos [5, 12]. Ths studis found that squntial accss, usually of th nti l, is th majo fom of accss. Supcomput l accss pattns (a scintic wokload) involv hug ls (tns to thousands of mgabyts) accssd pimaily squntially, somtims patdly [11]. Paalll l accss has bn discussd by Cocktt [2], but h did not study an actual wokload. 3 Modls and Mthods 3.1 Achitctual Modls Ou achitctual modl is a multipl instuction stam, multipl data stam (MIMD) multipocsso. A subst of th poblms and many of ou poposd solutions (although not ou implmntation) may also apply to mssag-basd distibutd-mmoy achitctus. W psnt th disk subsystm with paalll, indpndnt disks. W assum an intlavd mapping of ls to disks, with blocks of th l allocatd oundobin to all disks in th systm. Th l systm handls th mapping tanspantly, managing th disks and all qusts fo I/O. Th is a l systm manag unning on ach pocsso. This spads th I/O ovhad ov all pocssos and allows th us of all pocssos fo computation, ath than sving a st of pocssos xclusivly fo I/O. 3.2 Wokload Modl Paalll l systms and th applications that us thm a not sucintly matu fo us to know what foms might b typical. Paalll applications may us pattns that a mo complx than thos usd by unipocsso vsions of th sam application. Th lack of a al paalll wokload mploying paalll I/O lads us to us a synthtic wokload in ou tsts, which captus such nuancs of al wokloads as squntiality, gulaity, and int-pocss intactions. W wok with l accss pattns, ath than disk accss pattns. That is, w xamin th pattn of accss to logical blocks of th l ath than physical blocks on th disk. Thus, w mak no assumptions of disk layout. Not also that th application is accssing cods in th l, which a tanslatd into accsss to logical l blocks by th intfac to th l systm. Th l systm intnals, which a sponsibl fo caching, s only th block accss pattn. In ou sach w do not invstigat ad/wit l accss pattns, bcaus most ls a opnd fo ith ading o witing, with fw ls updatd [5, 12]. W xpct this to b spcially tu fo th lag ls usd in scintic applications. Thus w consid ad-only pattns, usd to dmonstat th bnts of caching, and wit-only pattns, usd to invstigat dlayd-wit policis. All squntial pattns consist of a squnc of accsss to squntial potions. A potion is som numb of contiguous blocks in th l. Not that th whol l may b considd on lag potion. Th accsss to this potion may b squntial whn viwd fom a local pspctiv, in which a singl pocss accsss succssiv blocks of th potion. W call ths locally squntial accss pattns, o just local pattns. This is th taditional notion of squntial accss usd in unipocsso l systms. Altnativly, th pattn of accsss may only look squntial fom a global pspctiv, in which many pocsss sha accss to th potion, ading disjoint cods within th potion. W call ths globally squntial accss pattns, o just global pattns. If th fnc stings of all th pocsss a mgd with spct to tim, th accsss follow a (oughly) squntial pattn. Th pattn may not b stictly squntial du to th slight vaiations in th global oding. W us ight psntativ ad-only paalll l accss pattns. Fou of ths a local pattns, th a global pattns, and on is andom. Th squntial natu of th pattns imply a low at of data fncing, which is impotant fo caching. Th dtails of th squntiality a only impotant fo pftching. lw Local Whol l: vy pocss ads th nti l fom bginning to nd. It is a spcial cas of a local squntial pattn with a singl potion. lfp Local Fixd-lngth Potions: ach pocss ads many squntial potions. Th squntial potions hav gula siz, although at dint placs in th l fo ach pocss. lp Local Random Potions: lik lfp, but using potions of igula (andom) siz. Potions may ovlap by coincidnc. sg Sgmntd: th l is dividd into a st of nonovlapping contiguous sgmnts, on p pocss. gw Global Whol l: th nti l is ad fom bginning to nd. Th pocssos ad distinct cods fom th l in a slf-schduld od, so that globally th nti l is ad xactly onc. gfp Global Fixd-lngth Potions: (analogous to lfp) pocssos coopat to ad xd-siz squntial potions. 2
gp Global Random Potions: (analogous to lp) pocssos coopat to ad andom-siz squntial potions. nd Random: cods a accssd at andom. W us th psntativ wit-only paalll l accss pattns. Two of ths a local pattns and on is a global pattn. lw1 A singl pocss wits th nti l fom stat to nish. Th oth pocsss a idl. sg Th l is dividd into disjoint sgmnts, on p pocss, and ach pocss wits its sgmnt fom stat to nish. gw Lik its ad-only countpat, this pattn wits cods of th l in a slf-schduld od. Not that ths pattns a not ncssaily psntativ of th distibution of th accss pattns actually usd by applications. W fl that this st covs th ang of pattns likly to b usd by scintic applications. 3.3 Mthods Ou mthodology is xpimntal, using a mix of implmntation and simulation. W implmntd a l systm tstbd calld RAPID-Tansit (\Rad- Ahad fo Paalll Indpndnt Disks") on a BBN GP00 Butty paalll pocsso [1], an MIMD machin. Sinc th multipocsso dos not hav paalll disks, thy a simulatd. Th tstbd is a havily paamtizd paalll pogam, incopoating th synthtic wokload (th application), th l systm (intfac and manag), and th st of simulatd disks. Th l systm allocats and manags a bu cach to hold disk blocks, dscibd blow. Th tstbd gaths statistics on many aspcts of th pfomanc of th l systm. This implmntation of th policis on a al paalll pocsso, combind with al-tim xcution and masumnt, allows us to dictly includ th cts of mmoy contntion, synchonization ovhad, int-pocss dpndncis, and oth ovhad, as thy a causd by ou wokload und vaious managmnt policis. This mthod allows us to valuat whth pactical caching policis can b implmntd. S [8] fo mo dtails. In this sction w dscib on simpl placmnt policy, which dtmins th blocks to plac whn a f bu is ndd, and sval wit policis, which dtmin whn nw data a wittn back to disk. Bu Rplacmnt Policy: W associat an instanc of th cach with a paticula opn l, caching th logical blocks of th l ath than th physical blocks of th disk. This is a shad cach concuntly svicing th qusts of all pocsss within a paalll application. Th wokload plays a signicant ol in dtmining th appopiat cach policis. Scintic applications oftn ad and wit sval mgabyts o gigabyts of data, gnally squntially [11]. Fo a cach to succd, th wokload must xhibit som locality. Tmpoal locality, wh cntly usd data will b usd again soon, is not psnt whn lag ls (much lag than th cach siz) a accssd squntially, vn if th ls a accssd patdly. Spatial locality, wh oth data na o in a cntly accssd block will b accssd soon, is a stong componnt of squntial accss pattns. Th combination of ths obsvations lads to a \toss-immdiatly" placmnt policy, wh only th most cntly usd (MRU) block mains in th cach. This is mo appopiat than th taditional LRU policy [17] (although of cous it is idntical to LRU with a stack siz of on). In th accss pattns w xpct to s in paalll scintic applications, anoth fom of locality occus. With intpocss locality, a block usd by on pocss is usd soon by anoth pocss (whn, fo xampl, ach is ading dint small cods fom th sam block). W xtnd th toss-immdiatly statgy to paalll accss pattns as follows: any block that is not th MRU block of any pocss may b placd. Thus th cach must hav at last as many bus as pocsss. Ou policy has many advantags. It nsus that th MRU block of ach pocss mains in th cach until that pocss has claly nishd with it. This is impotant, bcaus locality maks it likly that th pocss will us its MRU block again. If th w only on global MRU block, toss-immdiatly would plac som blocks still in us. If th w a global LRU policy, which had a singl LRU stack, an activ pocss could us many blocks, aticially aging th blocks of lss-activ pocsss and thus focing thm out. Finally, ous is simpl to implmnt: ach bu has a count in shad mmoy indicating th numb of pocsss that consid this block to b thi MRU block. Thus, intpocss locality is dictly includd. Whn th count achs zo, th block is f fo placmnt. If th block is dity (containing data not yt wittn to disk), th block must b wittn to disk and th disk wit compltd bfo th bu may b -usd. Bus that a availabl fo placmnt a kpt in a global f list. Wit Policis: A cach can impov l-wit pfomanc with wit-bhind, wh data is wittn into a bu, allowing th application to continu whil th bu is wittn to disk. If th disk wit is not initiatd immdiatly, it is tmd \dlayd witback," which taditionally has sval advantags: Som data disappas bfo it is wittn to disk (by bing ovwittn o by moval o tuncation of th l containing th data), and thus disk load is ducd. This is not likly in ou wokload. Busts of wit activity can b absobd by a cach, asynchonously witing th data to disk whil th application continus. Wh th is spatial locality (.g., whn multipl l wits a mad to th sam block), caching avoids multipl wits to th disk. This is of pim intst whn th is also intpocss locality involvd. 3
Th wit policy dtmins whn th \dity" bus a \cland" (wittn to disk). If a dity bu is wittn too lat, th cach lls with dity blocks and pocsss must idl waiting fo bus to b cland. If a dity bu is wittn too aly, costly mistaks may b mad. Th a two typs of mistaks possibl in wit-only accss pattns: ad and wit. If th application wits to a bu aft th bu has bn wittn to disk, th disk wit was a wit mistak. If th application wits to a block that has alady bn ushd fom th cach, causing th block to b ad back fom disk, th xta wit and ad is a ad mistak. A tchniqu that is appopiat fo a singl-pocss squntial accss pattn is to wit a block whnv th pocss movs on to th nxt block (o, if you tack th l point cafully, whn th pocss wits th last byt in th block). This tchniqu assums squntial accss: onc a block is wittn by th pocss, it will not b wittn. In a multipocss application with intpocss locality, howv, th actions of any on pocss do not claly indicat whn a block is complt. Fom th assumption of squntiality, howv, vy byt of th l (and hnc of any block in th l) is wittn xactly onc. Thus it is saf to wit th block to disk whn all byts of th block hav bn wittn. This lads dictly to ou WitFull policy blow. W implmntd sval distinct wit policis: WitThu, th simplst schm, focs a disk wit on vy l wit qust fom th application. This is idal fo blocks accssd only onc. WitBack dlays th disk wit until th bu is ndd fo anoth block. WitF issus a disk wit whn th bu nts th f list. Thus, it issus a wit bfo th bu is ndd fo -us, but aft it is no long in us by som pocsso. This is a compomis btwn WitThu and WitBack. WitFull issus th disk wit whn th bu is \full," dnd to b whn th numb of byts wittn to th bu is xactly qual to th siz of th bu in byts. 4 Expimnts W st biy dmonstat th nd fo a cach, and thn xamin th capabilitis of th fou wit policis. 4.1 Expimntal Paamts In all of ou xpimnts, w x most of th paamts and thn vay on o two paamts at a tim. Th paamts dscibd h a th bas fom which w mak oth vaiations. Each combination of paamts psnts on tst cas. Th w 20 pocsss unning on 20 pocssos. Th pattns all accssd 4 MByts of data, dividd up fo local pattns as 200 KByts p pocss. Th cach block siz was always 1 KByt, and th cod siz was usually on block (in on st of tsts w xpimnt with oth cod sizs). Not that in most pattns this tanslats to 4000 blocks ad fom (o wittn to) th disk, but in lw only 200 distinct blocks a ad sinc all pocsss ad th sam st of 200 blocks. Th cach containd 80 on-block bus. W also had th capability to tun th cach o, so all qusts wnt to th disk with no cach ovhad. Aft ach cod was accssd, dlay was addd in som tsts to simulat computation; this dlay was xponntially distibutd with a man of 30 msc. All oth tsts had no dlay aft ach accss, simulating an I/O-intnsiv pocss. Th l was intlavd ov 20 disks, at th ganulaity of a singl block. Disk qusts w quud in th appopiat disk quu. Th disk svic tim was simulatd using a constant aticial dlay of 30 msc, a asonabl appoximation of th avag accss tim in cunt tchnology fo small, inxpnsiv disk divs of th kind that might b plicatd in lag numbs on a multipocsso systm. 4.2 Masus Th RAPID-Tansit tstbd cods many statistics intndd to masu and intpt pfomanc. Th pimay pfomanc mtic fo masuing th pfomanc of an application is th total xcution tim. This, and all tim masus in th tstbd, is al tim. Total xcution tim incopoats all foms of ovhad (such as mmoy contntion, ad mistaks, tc.) and unxpctd cts, and thus it is th bst masu of ovall pfomanc. A not on th data: Evy data point in ach plot psnts th avag of v tials. Th cocint of vaiation (cv) is th standad dviation dividd by th man (avag). Fo all xpimnts in this pap, th cv was lss than 0.065 (usually much lss), maning that th standad dviation ov v tials was lss than 6.5% of th man. In ach tabl and plot w giv th maximum cv of all data points involvd. Th Idal Excution Tim: W compa th xpimntal xcution tim to a simpl modl of th idal xcution tim. Th total xcution tim is a combination of th computation tim, th I/O tim, and ovhad. In th idal situation, th is no ovhad, and ith all of th I/O is ovlappd by computation o all of th computation is ovlappd by I/O. Thus, th idal xcution tim is simply th maximum of th I/O tim and th computation tim. This assums that th wokload is vnly dividd among th disks and pocssos and that th disks a pfctly utilizd. No al xcution of th pogam can b fast than th idal xcution tim. With th bas paamt valus, both th I/O and th computation tims a 6 sconds, and thus th idal xcution tim is also 6 sconds. Th idal I/O tim fo lw is shot, only 0.3 sconds, sinc it only ads 200 blocks fom disk. Th idal computation tim fo lw1 with computation (and thus th idal xcution tim) is 120 sconds sinc th is only on pocsso involvd. 4
4.3 Caching Using th tstbd, w an all of ou accss pattns with and without caching. Ou point is not to dmonstat th supioity of ou paticula buplacmnt policy, but to dmonstat th basic bnt of a cach (fom tmpoal and spatial locality). W also hop to dtmin th cts of intpocss locality. Th cach, whn usd, containd 80 on-block bus. Th was no computation involvd in ths accss pattns. Th following tabl shows th sults of xpimnts on ou full st of ad-only accss pattns. With on-block cods, th was actually a slight pfomanc dgadation du to caching ovhad. Th was no impovmnt bcaus most of ths pattns did not fnc data in th cach (i.., th was no tmpoal locality). Som pattns (lp, gp and nd) mad som fncs, but so aly that thy w insignicant. Th lw pattn had many fncs (intpocss tmpoal locality), but xcution tim did not impov with caching bcaus all pocsss ad th sam block almost simultanously, and usd only on disk at a tim. Thus intpocss locality was impotant, but not bncial h. Th situation changd signicantly whn th cod siz was on-quat block. Excpt in th nd pattn, ach block was fncd fou tims, onc fo ach quat-block cod in th block. Without a cach, th block was ad fou tims fom th disk. With a cach, spatial locality (in th local pattns) and intpocss spatial locality (in th global pattns) was usd to avoid wasting disk bandwidth. (Not that th bnts would b lag fo small cod sizs, and signicant fo all non-intgal cod sizs.) Bcaus of th intpocss locality in th global accss pattns, howv, fou pocsss waitd fo ach fou-cod block to b ad fom th disk, and thus only onfouth of all disks w in us at any tim. Pftching can avoid this undutilization; s [9, 8, ] fo futh study of ad-only pattns and pftching. Rad-only pattns Total xcution tim, in sconds (cv < 0:038) On-block Quat-block Pattn No Cach Cach No Cach Cach lfp 6.3 6.7 24.6 7.1 lp 8.3 8.4 41.0 8.6 lw 6.9 7.1 36.5 7.3 sg 6.9 7.1 36.5 7.5 gfp 6.3 6.5 60.4 25.5 gp 6.4 6.7 60.4 25.4 gw 6.3 6.5 60.4 25.4 nd.6.7 41.1 40.8 Th nxt tabl shows th sults of xpimnts on ou wit-only accss pattns. H w compad th simpl WitBack caching policy with not caching. Sction 4.4 compas wit policis. Caching was fast in gw, sinc th dlayd wit allowd som ovlap btwn ovhad and I/O. Th lw1 pattn was most impovd bcaus, with dlayd wits, this on-pocsso pattn was abl to us mo than on disk. This is an xampl of a cach's ability to hlp applications us paalll disk bandwidth. Expimnts with quat-block cods dmonstat th al pow of caching: without a cach, all wits to a disk block aft th st wit had to ad th block fom th disk, updat th block, and wit th block back to disk (a ad mistak). With n cods p block, a cach ducd th 2n? 1 disk accsss p block to on p block. Wit-only pattns Total xcution tim, in sconds (cv < 0:015) On-block Quat-block Pattn No Cach Cach No Cach Cach lw1 127.3 16.4 853.1 55.7 sg 6.9 7.2 63.3 7.7 gw 6.3 6.1 3.0 8.7 4.4 Wit-Policy Expimnts W dsignd a st of xpimnts to valuat th ctivnss of ou wit policis acoss vaiations in wokload and cach siz. Ths xpimnts sk to answ th following qustions: What is th ct of cach siz? Is a lag cach usful? How do th policis act to th cod siz? In paticula, how do thy handl th intpocss locality in gw? Which (if any) policy is th most gnally succssful? Can a smat wit-buing policy hlp an application to btt us th availabl paalll I/O bandwidth? Cach-siz Vaiation: In ths xpimnts, th cach siz vaid fom 20 on-block bus to 200 onblock bus (1 to blocks p pocss). Th cod siz was on block, so ach block was accssd only onc. Not that WitFull and WitThu a inhntly quivalnt in ths accss pattns, bcaus th bu is full whn it is st wittn. In gw with computation, shown in Figu 1, WitBack was claly slowst, sinc it dlayd th disk wit too long. WitF is also slow than WitThu o WitFull. This is bcaus WitF dlays th disk wit fo a full MRU block until th nxt l systm accss, which is aft th pocss's comput cycl (without computation, WitF is simila to WitThu and WitFull). This dlay was too long, slowing down ovall xcution. Not that btwn 40 and 80 bus w th maximum usful cach siz. Foty bus cosponds to two bus p pocss, which allowd on to b lld whil th oth is wittn to disk. Th sults fo gw without computation giv simila conclusions. Th lw1 pattns an mo slowly than th gw pattns, bcaus on pocss could not div all 20 disks at full cincy (Figu 2). WitBack was much wos than th oth mthods, and WitF again was slow with computation. Lag cachs bntd th lw1 pattn by allowing mo disk paalllism to b usd. Th wit-only sg pattns had a dicult disk accss pattn (all pocsss bgan on th sam disk). A lag cach hlpd to allviat th sulting disk contntion, as sn in Figu 3, sinc th lag cach allowd pocsss to continu witing vn whn som disks w ovloadd. In ct, lag cachs allowd 5
Total Tim (sc) 14 12 8 6 gw with computation WitBack WitFull WitF WitThu idal 40 80 120 160 200 Cach siz (blocks) Total Tim (sc) Cach-siz vaiation fo lw1 19 WitBack 18 WitFull WitF 17 WitThu idal (6 sconds) 16 15 14 13 40 80 120 160 200 Cach siz (blocks) Figu 1: Cach-siz vaiation. Figu 2: Cach-siz vaiation. a long piplin to fom, using mo disks concuntly than with a shot piplin. This is spcially impotant as pocsso spds incas lativ to disk spds. This is an xcllnt xampl of th ability of a wll-managd cach to hlp a simpl-mindd pogam accss th potntially high bandwidth of paalll disks. Th sults fo sg with computation a not shown sinc thy o no nw insights. Fom ths sults, both WitThu and WitFull (ssntially quivalnt h) appa to b good witbuing mthods, in that thy had th bst ovall pfomanc. In som cass a lag cach was ndd to absob disk contntion poblms (as in sg) o a high wit qust at (as in gw without computation), but gnally two bus p pocss w sucint. Fo th xpimnts in th nxt sction w chos an 80-block cach (fou bus p pocss) bcaus that was a asonabl compomis fo all wokloads, basd on th sults in this sction. Rcod-siz Vaiation: In ths xpimnts w vaid th cod siz of th accss pattn with a xd cach siz of 80 on-block bus. Th total amount of data wittn, in blocks, was xd. Th vaiation includs both intgal and non-intgal cod sizs (lativ to th block siz). Th latt a impotant bcaus thy caus multipl accsss to many blocks, which should claly dintiat WitThu and WitFull. Figu 4a shows th cod-siz vaiation fo th wit-only gw accss pattn. WitThu is claly a poo choic fo small cod sizs, du to a hug numb of wit mistaks. WitF was smat, waiting until th bu was mostly unusd bfo issuing a disk wit, but it was still not pfct du to som mistaks and to not immdiatly witing th blocks to disk whn thy nally w ady to b wittn. Th dips occu bcaus th can b no mistaks with intgal cod sizs. WitBack was somtims fast than WitF bcaus it had fw wit mistaks. Finally, th WitFull mthod had a naly pfct 6- scond xcution tim ov all cod sizs, bcaus it issud th wit pcisly whn th block was ady to go to disk, and mad no mistaks. Th sults fo lw1 a shown in Figu 4b. Th high xcution tims w du to ducd I/O paalllism, bcaus (du to ovhad) on pocss could not kp 20 disks busy, vn with an 80-block cach. With non-intgal cod sizs this ovhad was incasd du to patd accsss to som blocks. Thus, th tim vais widly fo non-intgal cod sizs. Othwis, th sults a no supis: WitBack was usually slowst, and WitThu also slow fo small non-intgal cod sizs. Th cod-siz vaiation fo th sg pattn (Figu 4c) shows that WitThu was slowst, du to wit mistaks. Bcaus of th squntial accss pattn on ach pocsso, non of th oths had wit mistaks, and non had ad mistaks. Thus, cod siz was an impotant facto in th pfomanc of ou wit mthods. Fo intgal cod sizs, all mthods w ssntially indpndnt of cod siz. Fo non-intgal sizs, all but WitFull mad many mistaks. WitFull was thus th most gnally succssful wit policy. 5 Conclusion A lativly simpl cach managmnt statgy, basd on toss-immdiatly, povidd cint and ffctiv caching fo ou wokload. Most impotantly, it was an ctiv bas fo studying wit policis fo 6
Total Tim (sc) 20 15 5 a) Rcod-siz vaiation fo gw WitBack WitFull WitF WitThu idal 0 0 1 2 3 4 5 6 7 8 9 Rcod siz (blocks) Total Tim (sc) 60 50 40 30 20 b) Rcod-siz vaiation fo lw1 WitBack WitFull WitF WitThu idal 0 0 1 2 3 4 5 6 7 8 9 Rcod siz (blocks) Total Tim (sc) 30 25 20 15 5 c) Rcod-siz vaiation fo sg WitBack WitFull WitF WitThu idal 0 0 1 2 3 4 5 6 7 8 9 Rcod siz (blocks) Figu 4: Rcod-siz vaiation fo all th wit pattns. 7
Total Tim (sc) 15 14 13 12 11 9 8 7 6 Cach-siz vaiation fo sg WitBack WitFull WitF WitThu idal 40 80 120 160 200 Cach siz (blocks) Figu 3: Cach-siz vaiation. wit-only pattns. Caching was oftn abl to us locality, including intpocss locality, to hlp applications us th paalll disk bandwidth. In applications wh caching could not b xpctd to hlp, th cach ovhad causd a slight (though tolabl) slowdown. Givn th typs of wit-only accss pattns w xpct to b common in scintic wokloads, ou xploation of fou mthods shows that WitFull, th most sophisticatd of th mthods, was consistntly at o na th bst pfomanc in all situations. A faily small cach (40{80 blocks, i.., 2{4 blocks p pocss) was sucint to obtain th bst pfomanc, xcpt in th sg pattn, wh lag cachs hlpd mask th disk contntion. Lag cachs w thus only usful whn th was high disk contntion. (Although w did not study busty I/O, lag cachs should also b usful fo absobing busts of wit activity.) Rfncs [1] BBN Advancd Computs. Butty Poducts Ovviw, 1987. [2] Thomas W. Cocktt. Fil concpts fo paalll I/O. In Pocdings of Supcomputing '89, pags 574{579, 1989. [3] Pt Dibbl, Michal Scott, and Cala Ellis. Bidg: A high-pfomanc l systm fo paalll pocssos. In Pocdings of th Eighth Intnational Confnc on Distibutd Comput Systms, pags 154{161, Jun 1988. [4] Pt C. Dibbl. A Paalll Intlavd Fil Systm. PhD thsis, Univsity of Rochst, Mach 1990. [5] Rick Floyd. Shot-tm l fnc pattns in a UNIX nvionmnt. Tchnical Rpot 177, Dpt. of Comput Scinc, Univ. of Rochst, Mach 1986. [6] Jams C. Fnch, Tnc W. Patt, and Miganka Das. Pfomanc masumnt of a paalll input/output systm fo th Intl ipsc/2 hypcub. Pocdings of th 1991 ACM Sigmtics Confnc on Masumnt and Modling of Comput Systms, pags 178{187, 1991. [7] Michll Y. Kim. Synchonizd disk intlaving. IEEE Tansactions on Computs, C-35(11):978{ 988, Novmb 1986. [8] David Kotz. Pftching and Caching Tchniqus in Fil Systms fo MIMD Multipocssos. PhD thsis, Duk Univsity, Apil 1991. Availabl as tchnical pot CS-1991-016. [9] David Kotz and Cala Schlatt Ellis. Pftching in l systms fo MIMD multipocssos. IEEE Tansactions on Paalll and Distibutd Systms, 1(2):218{230, Apil 1990. [] David Kotz and Cala Schlatt Ellis. Pactical pftching tchniqus fo paalll l systms. In Fist Intnational Confnc on Paalll and Distibutd Infomation Systms, Dcmb 1991. To appa. [11] Ethan Mill. Input/Output bhavio of supcomputing applications. Tchnical Rpot UCB/CSD 91/616, Univsity of Califonia, Bkly, 1991. Submittd to Supcomputing '91. [12] John Ousthout, Hv Da Costa, David Haison, John Kunz, Mik Kupf, and Jams Thompson. A tac divn analysis of th UNIX 4.2 BSD l systm. In Pocdings of th Tnth ACM Symposium on Opating Systms Pincipls, pags 15{24, Dcmb 1985. [13] David Pattson, Gath Gibson, and Randy Katz. A cas fo dundant aays of inxpnsiv disks (RAID). In ACM SIGMOD Confnc, pags 9{116, Jun 1988. [14] Paul Pic. A concunt l systm fo a highly paalll mass stoag systm. In Fouth Confnc on Hypcub Concunt Computs and Applications, pags 155{160, 1989. [15] Knnth Salm and Hcto Gacia-Molina. Disk stiping. In IEEE 1986 Confnc on Data Engining, pags 336{342, 1986. [16] Alan Jay Smith. Disk cach-miss atio analysis and dsign considations. ACM Tansactions on Comput Systms, 3(3):161{203, August 1985. [17] Michal Stonbak. Opating systm suppot fo databas managmnt. Communications of th ACM, 24(7):412{418, July 1981. 8