SINGLE SIMULATION CONFIDENCE INTERVALS USING THE DELTA METHOD

Transcription

1 Roser Chrstoph ad Masaru Nakao. Sgle Smulato Cofdece Itervals Usg the Delta Method. I Iteratoal Smposum o Schedulg edted b H. Fumoto ad M. Kuroda Hamamatsu Japa 00. SINGLE SIMULATION CONFIDENCE INTERVALS USING THE DELTA METHOD Chrstoph Roser Masaru Nakao Toota Cetral Research ad Developmet Laboratores Nagakute Ach JAPAN croser@robotcs.ttlabs.co.p akao@robotcs.ttlabs.co.p ABSTRACT Ths paper descrbes a method for the calculato of cofdece tervals of smulato throughputs ad utlzatos. The method s based o the delta method ad uses ol a sgle smulato where the varato of the uderlg meas s used to determe the varato of the performace fucto b usg the frst dervatve of the performace fucto. Whle the delta method requres depedet dstrbuted data such data s frequetl avalable for ma performace measures allowg the practcal applcato of the method. I addto the method ca also be used for short smulatos or rare evet applcatos where methods based o batch meas fal. Ths method ca easl be mplemeted to estg smulato software. INTRODUCTION Schedulg s the process to arrage a umber of tasks a sequece. A frequet goal s to reduce the overall tme for the performace of all tasks or to esure the completo of some tasks before a deadle. The tme eeded for the tasks have to be kow to esure the tmel completo of the tasks. Ufortuatel these tmes are rarel statc but var depedg o outsde flueces ad radom evets. Ofte smulato s used to estmate these tmes ad cofdece tervals are used to determe the accurac of these estmates. The uderlg equatos to calculate a cofdece terval are well kow (Devore 995). However there are some complcatos to calculate cofdece tervals dscrete evet smulato. Oe complcato s that depedet ad detcall dstrbuted (..d.) data s requred but smulato data s frequetl ether depedet (e.g. watg tmes) or detcall dstrbuted (e.g. warmg up perod) (Re 997) (Klee 987). Addtoall ma performace measures dscrete evet smulato are a fucto of oe or more meas. For eample the throughput s the verse of the mea tme betwee completos of two parts. Ths further complcates the calculato of cofdece tervals Ths paper descrbes a method to determe the varace of the fucto of the meas of oe or more varable usg the mea ad varace of the varables ad the gradet of the fucto at the mea value. (Aleopoulos ad Sela 000) (Law ad Kelto 99) ad (Baks 998) gve good overvews of the curretl avalable methods for cofdece terval calculato smulatos wth the most popular method beg the batch meas method. There are a umber of feret batch meas ad related methods developed. (Sela 99) (Goldsma 99) (Schmeser ad Sog 996) (Pawlkowsk 990) gve a overvew of feret batchg methods lke overlappg batch meas o overlappg batch meas or fed umber of batches methods. There are a umber of problems assocated wth the batch mea method. Frst t s fcult to decde o the umber of batches. Secodl large data sets are eeded to acheve vald cofdece tervals. Thrd feret batchg methods fer wdel ther results. Fall t s computatoall tesve to calculate the cofdece tervals ad therefore the cofdece tervals are usuall ot calculated cotuousl durg smulatos. Ths paper addresses the calculato of cofdece tervals of fuctos of mea values. The preseted method avods most of the above problems for..d. data. Eample applcatos are gve for throughputs ad utlzatos. The method s the valdated epermetall usg a comple smulato. THE DELTA METHOD The delta method calculates the devato of a fucto of oe or more meas based o the mea ad devato of the fucto varables usg the gradets of the fucto at the mea values. The followg secto descrbes the delta method leadg to the geeral equato for ths method (3). Assume there s a geeral performace measure as a fucto f of the mea values oe or more varables as for eample the throughput s a verse of the tme betwee parts or the utlzato s the workg tme dvded b the tme betwee parts. The mea values are calculated based o a set of data values where the mea ad the stadard devato σ s calculated usg the well-kow equatos as show ().

2 Roser Chrstoph ad Masaru Nakao. Sgle Smulato Cofdece Itervals Usg the Delta Method. I Iteratoal Smposum o Schedulg edted b H. Fumoto ad M. Kuroda Hamamatsu Japa 00. () Yet f the mea values are appled to the fucto f ol oe performace measure s geerated. The varato of the performace measure ad subsequetl the cofdece terval s et ukow. Whle t s possble to eter the dvdual values to the equato f the resultg mea ad varato of the performace measure would be correct as show equato () for all olear fuctos.e. the fucto of the mea would fer from the epected value of the fucto of the dvdual data values. Ol for lear fuctos f wll the fucto of the meas ad the mea of the fucto be equal (Papouls 99). f E f () The delta method replaces the fucto f b ts taget f * at the mea values. Usg ths taget f * t s possble to determe the stadard devato σ f of the fucto f of the meas based o the devato of the varables σ. Fgure vsualzes the throughput eample for a tagetal le f * replacg the fucto f. Tme per Part Throughput =/ E[f()] Taget Fgure : TANGENT AT THE MEAN VALUE Usg the stadard devato ad the covarace of the varables the stadard devato of the fucto value ca be determed usg the delta method as show equato (3) (Rao 00). Equato (3) cludes the effect of the correlato betwee two pared varables where cov[ ] s the ubased estmate of the covarace as show equato (4) (Papouls 99). Cov d d d d (3) Cov (4) The resultg stadard devato σ of the fucto value ca the be used to calculate desred measures of accurac as for eample a cofdece terval as show equato (5) where t s the studet-t dstrbuto ad α s the cofdece level (Studet 908).

3 Roser Chrstoph ad Masaru Nakao. Sgle Smulato Cofdece Itervals Usg the Delta Method. I Iteratoal Smposum o Schedulg edted b H. Fumoto ad M. Kuroda Hamamatsu Japa 00. CI t / (5) Ths approach s vald f the uderlg data s..d. The depedece ca be tested usg the vo Neuma rato η of the mea squared successve ferece to the varato (RMSSDV) (Neuma 94; Neuma 94). Equato (6) shows the calculato of the RMSSDV η based o a set of data of sze where the mea squared ferece betwee successve data s dvded b the varace of the data. Varats of equato (6) ca be foud (Klee 987) or (Steger ad Wlso 999). (6) If the data s depedet the RMSSDV η has a value of two. Thus ths method ca be used to determe f the collected data s appromatel depedet (.e. wth a mea value at or ear two) or ot (.e. the mea fers from two). COMMON PERFORMANCE MEASURES Frequeces ad Throughputs Frequeces are a measuremet of the umber of occurreces a gve tme. Throughputs are a tpe of frequeces measurg the umber of parts produced a gve perod of tme. Other frequeces are for eample falure rates.e. the umber of falures a gve perod of tme. These frequeces are defed b the umber of occurreces of a evet a gve perod of tme. Ths ca also be descrbed as the verse of the mea tme betwee the occurreces of a evet. Subsequetl the frequec ca be defed as the verse of the average tme betwee occurreces as show equato (7). The advatage of ths approach s that the devato of the tme betwee occurreces ca be calculated ad therefore also the devato of the frequec. f (7) Applg equato (3) to the fucto of the frequec equato (7) the stadard devato of the frequec ca be determed as show equato (8). A subsequet cofdece terval ca be calculated as show equato (5). (8) Percetages Aother commo performace measure dscrete evet smulato are percetages of tmes as for eample the percetage of tme a mache s workg or the percetage of tme a mache s uder repar. I geeral a percetage ca be calculated b dvdg the total tme a mache s a certa state b the total smulato tme. Ths ca also be represeted as the mea durato a mache s a certa state dvded b the mea durato betwee the beggs of a certa state. For eample the percetage repar s the mea tme to repar dvded b the mea tme betwee the beggs of repars. The fucto of the two mea values s show equato (9). f (9) Applg equato (3) to the fucto of the percetage equato (9) the stadard devato of the percetage ca be determed as show equato (0). A subsequet cofdece terval ca be calculated as show equato (5). Cov (0) 3

4 Roser Chrstoph ad Masaru Nakao. Sgle Smulato Cofdece Itervals Usg the Delta Method. I Iteratoal Smposum o Schedulg edted b H. Fumoto ad M. Kuroda Hamamatsu Japa 00. COMPLEX MANUFACTURING SYSTEM The preseted method was verfed usg a comple smulato eample cosstg of seve maches a comple settg ad a mture of two feret products. The smulato was performed usg the GAROPS smulato software as show Fgure (Kubota Sato ad Nakao 999) (Nakao et al. 994). Fgure : GAROPS Smulato Eample The total smulato tme was almost two ears of smulato. After removg the warmg up perod ths data was the splt to 0 subsets wth a smulato tme of 6 das each. I order to calculate vald cofdece tervals the data has to be depedet. Therefore the RMSSDV has bee calculated for the data usg equato (6) to determe f the data s depedet. Whle smulatos are otorous for depedet data the actual mache performace data was surprsgl ofte depedet or ear depedet. Despte the comple teractos of the sstem most mache performace measures were depedet. I fact out of 46 measured parameters as for eample the workg tmes or the tme betwee falures all but four were appromatel depedet wth a RMSSDV η betwee.7 ad.. Ths allows the calculato of a vald stadard devato ad a cofdece terval for these values as descrbed above. For each of the 0 subsets the frequeces ad the percetages of all maches workg dle blocked or repared were measured ad the 95% cofdece tervals calculated. These cofdece tervals were the compared to the overall average whch are ver close to the ukow true value. Ideall for cofdece tervals wth a cofdece level of 95% 95% of the cofdece tervals cota the true value.e. the desred coverage s 95%. However the real case the percetage of the cofdece tervals cotag the true value ma fer from the deal case.e. the actual coverage fers from the desred coverage. The closer the actual coverage s to the desred coverage the more accurate s the cofdece terval method. Table shows a overvew of the coverage results of the comple smulato. Table : SIMULATION EXAMPLE COVERAGE Performace Desred Actual Too Too Measure Coverage Coverage Small Large Frequec 95% 94.4%.9%.7% Percetage 95% 9.9% 4.3%.8% Out of the 69 frequec cofdece tervals wth a desred coverage of 95% the actual coverage was 94.44%. The staces where the log-term average was outsde of the cofdece terval were also smmetrcall dstrbuted wth.8% uder predcto ad.7% over predcto. Ths dcates a ver good overall ft. Out of the 69 percetage cofdece tervals wth a desred coverage of 95% the actual coverage was 9.86%. The staces where the log-term average was outsde of the cofdece terval cotaed 4.3% uder predcto ad.8% over predcto. Whle the ft s ot as good as for the frequeces the coverage s stll ver close to the desred coverage. Overall the actual coverage s almost detcal wth the desred coverage. Furthermore the actual coverage s also cel cetered wth the umber of over ad uder predctos beg almost equal. The preseted method has bee compared to the batchg method where the cofdece terval s based o the batch meas. The cofdece tervals of the frequeces ad percetages have bee obtaed from 00 smulatos usg a fed umber of 30 batches wth depedet batch meas. A total of 80 cofdece tervals for both the frequeces ad percetages have bee evaluated of whch ol 498 ad 503 cofdece tervals cotaed the true mea value. Therefore the batch meas method had coverage of o- 4

5 Roser Chrstoph ad Masaru Nakao. Sgle Smulato Cofdece Itervals Usg the Delta Method. I Iteratoal Smposum o Schedulg edted b H. Fumoto ad M. Kuroda Hamamatsu Japa 00. l.8% ad 68.8% for the frequeces ad throughputs respectvel mssg the desred coverage of 95 b a wde marg ad s clearl feror to the delta method for depedet data. CONCLUSION I cocluso the method provdes ver accurate results for ear depedet ad detcall dstrbuted data. Whle smulato data s kow to be depedet the mache performace data was actuall foud to be frequetl depedet allowg the calculato of the cofdece tervals usg the delta method. Compared to batchg t s ver fast to calculate the cofdece terval as t s ot ecessar to calculate feret batch szes ad perform comple statstcal tests. Moreover f addtoal data becomes avalable ths data ca easl be tegrated to the prevous calculato ad the cofdece terval ca be updated. Ths allows a sequetal addg of data whle updatg the cofdece terval. Furthermore the method works also wth small sets of data. Ths s etremel useful for eample to aalze rare evets where eve a log smulato does ot have ma occurreces of the rare evet ad subsequetl batch meas methods caot be appled. I summar the method provdes a preferable alteratve to calculate the cofdece tervals for appromatel depedet data. REFERENCES Aleopoulos Chrstos ad Sela Adrew F Output Aalss for Smulatos. I Wter Smulato Coferece J. A. Joes R. R. Barto K. Kag ad P. A. Fshwck 0-08 Orlado Florda USA. Baks Jerr 998. Hadbook of Smulato: Prcples Methodolog Advaces Applcatos ad Practce. Joh Wle & Sos. Devore Ja L Probablt ad Statstcs for Egeerg ad the Sceces. Belmot: Dubur Press Wadsworth Publshg. Goldsma Davd 99. Smulato output aalss. I Wter Smulato Coferece Arlgto VA USA. Klee Jack P. C Statstcal Tools for Smulato Practtoers. New York ad Basel: Marcel Dekker. Kubota Fumko Sato Shuch ad Nakao Masaru 999. Eterprse Modelg ad Smulato Platform Itegrated Maufacturg Sstem Desg ad Suppl Cha. I IEEE Coferece o Sstems Ma ad Cberetcs 5-55 Toko Japa. Law Averll M. ad Kelto Davd W. 99. Smulato Modelg & Aalss. McGraw Hll. Nakao Masaru Sugura Noro Taaka Moru ad Kuo Toshtaka 994. ROPSII: Aget Oreted Maufacturg Smulator o the bass of Robot Smulator. I Japa-USA Smposum o Fleble Automato 0-08 Kobe Japa. Neuma Joh vo 94. Dstrbuto of the Rato of the Mea Square Successve Dfferece to the Varace. Aals of Mathematcal Statstc Neuma Joh vo 94. A Further Remark Cocerg the Dstrbuto of the Rato of the Mea Square Successve Dfferece to the Varace. Aals of Mathematcal Statstc Papouls Athaasos 99. Probablt Radom Varables ad Stochastc Processes. McGraw-Hll. Pawlkowsk Krzsztof 990. Stead-state smulato of queueg processes: surve of problems ad solutos. ACM Computg Surves () Rao C. Radhakrsha 00. Lear statstcal ferece ad ts applcatos. Wle Press. Re Horst 997. Taschebuch der Statstk. Frakfurt am Ma: Verlag Harr Deutsch. Schmeser Bruce W. ad Sog Whemg Ta 996. Batchg methods smulato output aalss: what we kow ad what we do't. I Wter Smulato Coferece J. M. Chares D. J. Morrce D. T. Bruer ad J. J. Swa -7 Sa Dego CA USA. Sela Adrew F. 99. Advaced output aalss for smulato. I Wter Smulato Coferece Arlgto VA USA. Steger Natale Mller ad Wlso James R Improved Batchg for Cofdece Iterval Costructo Stead State Smulato. I Wter Smulato Coferece P. A. Farrgto D. T. Nembhard D. T. Sturrock ad G. W. Evas Phoe AZ USA. Studet 908. The probable error of a mea. Bometrka