Conformance Analysis of ASML s Test Process

Transcription

1 Conformanc Analysis of ASML s Tst Procss A. Rozinat 1, I.S.M. d Jong 2, C.W. Günthr 1, and W.M.P. van dr Aalst 1 1 Dpartmnt of Information Systms, Eindhovn Univrsity of Tchnology P.O. Box 513, NL-5600 MB, Eindhovn, Th Nthrlands {a.rozinat,c.w.gunthr,w.m.p.v.d.aalst}@tu.nl 2 ASML, P.O. Box 324, NL-5500 AH, Vldhovn, Th Nthrlands ivo.d.jong@asml.com Abstract. Procss mining allows for th automatd discovry of procss modls from vnt logs. Ths modls provid insights and nabl various typs of modl-basd analysis. Howvr, in many situations alrady som normativ procss modl is givn, and th goal is not to discovr a modl, but to chck its conformanc. Th procss mining framwork ProM provids a conformanc chckr abl to invstigat and quantify dviations btwn th ral procss (as rcordd in th vnt log) and th modld procss. Th conformanc chckr is on of th fw tools availabl today that is abl support rgulatory complianc, i.., nsuring that organizations and popl tak stps to comply with rlvant laws, rgulations, and procdurs. In this papr, w rport on a cas study whr th ProM framwork has bn applid to th tst procsss of ASML (th lading manufacturr of wafr scannrs in th world). In this cas study, w focus on th conformanc aspct and compar th tst procss as it is rally xcutd to th idalizd rfrnc modl that ASML is using to instruct thir tst tams. This rvald that th ral procss is much mor complicatd than th idalizd rfrnc procss. Morovr, w wr abl to suggst concrt improvmnt actions for th tst procss at ASML. Ky words: Procss mining, Conformanc chcking, Cas study 1 Introduction Corporat scandals hav triggrd an incrasd intrst in corporat govrnanc, risk managmnt, and rgulatory complianc. As a rsult nw rgulations such as th Sarbans-Oxly Act, Basl II, HIPAA, tc. wr introducd. Som of th ky lmnts ar: accountability, auditability, privacy, documntation, policy, and managability of information. In this papr, w focus on complianc. In particular, w focus on th qustion Do organizations and popl do what is documntd in procss modls?. To addrss this qustion w conduct a cas study whr w apply ProM s conformanc chckr [22] to on of th procsss of ASML. To position our work, w first introduc procss mining. Th basic ida of procss mining is to discovr, monitor and improv ral procsss by xtracting

2 2 Procdings of GRCIS 2009 knowldg from vnt logs. Today many of th activitis occurring in procsss ar ithr supportd or monitord by information systms. Considr for xampl ERP, WFM, CRM, SCM, and PDM systms to support a wid varity of businss procsss whil rcording wll-structurd and dtaild vnt logs. Howvr, also high-tch dvics such as X-ray machins, wb srvics, tc. rcord vnts. All of ths applications hav in common that thr is a notion of a procss and that th occurrncs of activitis ar rcordd in so-calld vnt logs. Assuming that w ar abl to log vnts, a wid rang of procss mining tchniqus coms into rach. Th basic ida of procss mining is to larn from obsrvd xcutions of a procss. Procss mining can b usd to (1) discovr nw modls (.g., constructing a Ptri nt that is abl to rproduc th obsrvd bhavior), (2) to chck th conformanc of a modl by chcking whthr th modld bhavior matchs th obsrvd bhavior, and (3) to xtnd an xisting modl by projcting information xtractd from th logs onto som initial modl (.g., show bottlncks in a procss modl by analyzing th vnt log). All thr typs of analysis hav in common that thy assum th xistnc of som vnt log. tst tam tst procss wafr scannr modls analyzs rcords vnts (procss) modl discovry conformanc xtnsion vnt logs Fig. 1. Basd on th vnt logs of th wafr scannrs, thr classs of procss mining tchniqus ar possibl: (1) discovry, (2) conformanc, and (3) xtnsion In this papr, w focus on conformanc chcking and apply ProM s conformanc chckr [22] to th tst procss of ASML. ASML is th world s lading manufacturr of chip-making quipmnt and a ky supplir to th chip industry. ASML dsigns, dvlops, intgrats and srvics advancd systms to produc smiconductors,.g., wafr scannrs that print th chips. Thr is an ongoing ffort to rduc th lin width on silicon wafr to nhanc th prformanc of th manufacturd smi-conductors. Evry nw gnration of wafr scannrs is balancing on th bordr of what is tchnologically possibl. As a rsult, th tsting of manufacturd wafr scannrs is an important but also tim-consuming procss. Evry wafr scannr is tstd in th factory of ASML. Whn it passs all tsts, th wafr scannr is disassmbld and shippd to th customr whr

3 Procdings of GRCIS th systm is r-assmbld. At th customr s sit, th wafr scannr is tstd again. Clarly, tsting is a tim-consuming procss and taks svral wks at both sits. Sinc tim-to-markt is vry important, ASML is involvd in an ongoing ffort to rduc th tst priod. To assist ASML in ths fforts, w applid procss mining tchniqus to thir tst procsss. Rathr than focusing on fault dtction, th subjct of study is hr th tst procss itslf. At any point in tim, ASML s wafr scannrs rcord vnts that can asily b distributd ovr th intrnt. Hnc, any vnt that taks plac during th tst procss can b rcordd asily. Th availability of ths vnt logs and th dsir of ASML to improv th tsting procss triggrd th cas study rportd in [24]. Using procss discovry, w trid to answr th qustion How ar th tsts actually xcutd?, i.., basd on th vnt logs w automatically constructd procss modls showing th ordring and frquncy of tst activitis. In this papr, w thn compard thm to th idalizd rfrnc modl. This rvald that th ral procss is much mor complicatd than th idalizd rfrnc modl that ASML is using to instruct th tst tams. Th rfrnc modl shows a rathr structurd procss whil in rality th tsting procss rquirs much mor flxibility. Using conformanc chcking tchniqus, w invstigatd this furthr by answring th qustion How compliant ar th actual tst xcutions to th rfrnc procss?. Through conformanc chcking w wr abl to quantify and pinpoint th dviations of th ral tst procss from th idalizd rfrnc modl. For th cas study w usd our ProM framwork 3. ProM is opn sourc and uss a plug-abl architctur,.g., dvloprs can add nw procss mining tchniqus by adding plug-ins without spnding any fforts on th loading and filtring of vnt logs and th visualization of th rsulting modls [1]. Vrsion 5.0 of ProM provids 230 plug-ins. For xampl, thr ar mor than 15 plug-ins to discovr procss modls from vnt logs. Th rmaindr of this papr is organizd as follows. Sction 2 rviws rlatd work both in procss mining and th tst procss optimization domains. Nxt, th contxt of th cas study is dscribd in mor dtail in Sction 3. Sction 4 prsnts th rsults of this study, and concrt improvmnt actions for th ASML tst procss ar proposd in Sction 5. Sction 6 concluds th papr. 2 Rlatd Work Sinc th mid-nintis svral groups hav bn working on tchniqus for procss mining [3, 4, 10], i.., discovring procss modls basd on obsrvd vnts. In [2] an ovrviw is givn of th arly work in this domain. Th papr builds on th conformanc chcking tchniqus prsntd in [22]. Ths tchniqus ar inspird by th fitnss function usd in gntic procss mining [20]. Also rlatd is th work by Cook [9, 8] whr th vnt strams of a procss modl and a log ar compard basd on string distanc mtrics. Procss mining can b sn in th broadr contxt of Businss Procss Intllignc (BPI) and Businss Activity Monitoring (BAM). In [14, 25] a BPI toolst 3 ProM can b frly downloadd from

4 4 Procdings of GRCIS 2009 on top of HP s Procss Managr is dscribd. Th BPI toolst includs a so-calld BPI Procss Mining Engin. In [21] Zur Mühln dscribs th PISA tool which can b usd to xtract prformanc mtrics from workflow logs. Similar diagnostics ar providd by th ARIS Procss Prformanc Managr (PPM) [18]. It should b notd that BPI tools typically do not allow for procss discovry and conformanc chcking, and offr rlativly simpl prformanc analysis tools that dpnd on a corrct a-priori procss modl [17]. In [5] it is suggstd that databas tchnology can play an important rol in assisting complianc with th intrnal control provisions of SOX. Most of th work on conformanc chcking has bn don on modl analysis without taking into account vnt logs. For xampl, in [13] it is chckd whthr businss procsss ar compliant with businss contracts, and in [12] a non-monotonic dontic logic of violations is usd to dtct all obligations that will not ncssarily b fulfilld by xcuting th modl. In [19], th authors introduc OPAL, a complianc-chcking framwork, and rlatd tools, including a static mthod to chck businss procss modls against complianc ruls. In [11] smantically annotatd procss modls and formal rprsntations of complianc rquirmnts ar compard for auditing BPMN procss modls for complianc with lgislativ/rgulatory rquirmnts, and for xploring altrnativ modifications to rstor complianc in th vnt that th procsss ar found to b non-compliant. Th conformanc chcking tchniqus usd in this papr ar gnric and can b applid to various typs of procsss. Hnc, it can b usd to analyz th logs of ERP, WFM, CRM, SCM, and PDM systms. Howvr, in this papr w apply our tchniqus to a particular typ of procss: tsting ASML s wafr scannrs. S [7, 6] for mor information on tsting and tst dsign in this particular stting. Th cas study rportd in [24] alrady xplors th applicability of procss mining to improv ASML s tst procss, and, for xampl, analyzs idl tims to shortn th tim-to-markt. In this papr, w invstigat th diffrncs btwn th actual, xcutd tst squncs and th plannd tst squncs in mor dtail. 3 Cas Study This sction introducs th cas study whr procss mining was applid to th tst procss of ASML s wafr scannrs. Aftr dscribing th tst procss of a wafr scannr in mor dtail (Sction 3.1), w look at th log data rcordd during ths tsts (Sction 3.2). Th vnt logs srv as input for our procss mining tchniqus and th rsults of thir analysis ar dscribd in Sction Th Tst Procss Th whol tst procss of a wavr scannr at ASML consists of thr phass: (1) th calibration phas, (2) th tst phas (th actual tsting), and (3) th final qualification phas. Th whol procss taks svral wks. Whn finishd, th wafr scannr is partly takn apart and shippd to a customr. A part of th

5 Procdings of GRCIS calibration and tst phas is rpatd at th customr sit, aftr r-assmbling th wafr scannr. Why is this tst procss so important for ASML? ASML oprats in a markt whr th tim-to-markt of systm nhancmnts and th tim-to-markt of nw systm typs is critical. Wafr scannrs ar continuously nhancd. As a rsult, th numbr of manufacturd wafr scannrs of a singl typ is typically lss than 50. With ach nw typ, parts of th calibration and tst phas ar adjustd. On avrag fiv diffrnt systm typs ar manufacturd in paralll. Th short tim-to-markt, th constant innovation, and th high valu of wafr scannrs mak tsting vry important. Spnding too much tim on tsting will rsult in high invntory costs and lost sals. Howvr, inadquat tsts will rsult in systms which ar malfunctioning. Fig. 2. Exampl squnc of thr job stps with a synchronization point Sts of calibration and tst actions ar groupd into so-calld job stps. Ths job stps ar xcutd according to a crtain squnc. Only larg changs in th systm dsign rsult in changs in th job stp squnc, so th job stp squnc can b considrd a fixd squnc across diffrnt systms. Som of ths jobstps can b xcutd indpndntly of ach othr. An xampl squnc of thr job stps is dpictd in Figur 2. Not that in ASML such structurs ar rfrrd to as squncs. Howvr, strictly spaking ths ar procss modls rathr than squncs. Th synchronization point sync nforcs that both job stp A and job stp B must b finishd bfor job stp C can. Each calibration action or tst cas can fail. Som of th causs for tst failur can rquir a rplacmnt of a faulty hardwar componnt. Th duration of this rplacmnt can tak up to hours or longr. If such a failing tst is in th xampl job stp A, thn th indpndnt job stp B can b d to nsur maximal progrss. Whn th rplacmnt hardwar bcoms availabl, ithr job stp B is finishd first and thn job stp A is finishd, or th othr way around. Job stp C is d whn job stp A and B ar both finishd. Not that a failur in a tst cas in job stp C rsults in no activity on th systm (idl tim) until th malfunctioning part of systm is fixd and tsting can continu.

6 6 Procdings of GRCIS 2009 Som of th causs for a failur can b fixd immdiatly. For xampl, som paramtrs in th systm can b out of spcification. This masurmnt information can now b usd to adopt th control st-points in th systm. Aftr a scond masurmnt, th paramtrs can b within spcification and th tst passs. Most of th softwar which xcuts th tsts is constructd such that this fast-fix loop is automatd. Tsting, calibration and rtsting is prformd in a singl tst. Finally, a chang in low-lvl machin paramtrs, bcaus of a hardwar rplacmnt, can caus a r-xcution of a prvious job-stp. For instanc, th profil of som of th mirrors in a wafr scannr ar masurd and stord in X,Y and Z dirctions. This profil information is usd in all positioning calibrations, such that th rrors causd by th non-flat mirrors ar minimizd. Rplacing ths mirrors rsults in a nw profil. For this rason, a larg st of job stps nds to b rdon if a faulty mirror is rplacd in on of th last job stps in th squnc. In summary, job stps ar xcutd according to a fixd squnc for a st of machin typs. Th squnc allows variation of th dtaild tsts within th limits of th synchronization points. Th actual xcution of tsts rsults in failing tst cass, which can rsult in a lngthy r-tst of parts of th squnc dpnding on th failur at hand. For ASML, th goal is to minimiz th waiting tim for a hardwar fix (idl tim) and to rduc th r-xcution of parts of th job-stp squnc. This goal could b asily mt by tsting all componnts and building blocks thoroughly bfor and during systm assmbly. Howvr, th incras in tst ffort would rsult in an incras of th total tst duration and thrfor an incras in tim-to-markt. This is th main rason that tsting vrything thoroughly bforhand is not considrd a solution, so th main goal is a rduction of th duration of th tst procss and not cutting costs. 3.2 Log Data and Convrsion Each wafr scannr in th ASML factory producs a log of th softwar tsts which ar xcutd. Th manual assmbly and calibration actions ar not loggd and appar as idl tim in this log. Th wafr scannr is calibratd and tstd using calibration and prformanc softwar, indicatd in th logging as a fourlttr cod. Th logging contains th and stop momnt of ach tst. Th idl tim, i.., th tim btwn stop of th prvious tst and th of th nxt tst, is not spcifid in dtail. This idl tim has a numbr of causs, ranging from inxprincd oprators rading th procdurs, th nd of th automatd tst quu during th night to diagnosing a problm, or waiting for additional parts. Som parts of th tst squnc ar xcutd in an automatd fashion. Th oprator s a tst quu which contains a st of tst cass which ar xcutd in a squnc. This tst quu can also contain part of th rcovry and rtry squnc for crtain failing tst cass. Th rcovry or rtry tsts ar xcutd dpnding on th outcom of a tst in th quu. An xampl fragmnt of th tst log of on of th wafr scannrs is dpictd in Figur 3(a). Each lin corrsponds to th xcution of on tst. Th numbr at th bginning of th lin idntifis th machin (i.., th wafr scannr) that is

7 Procdings of GRCIS , :33:13, :33:39,POLA 1596, :33:50, :34:46,OSWL 1596, :34:48, :35:10,OSSP 1596, :36:18, :36:49,AHZI 1596, :42:18, :43:25,DSNA 1596, :43:39, :44:56,AHZI 1596, :44:57, :59:10,SVEI 1596, :15:37, :33:25,SVEI 1596, :35:00, :53:24,SCEI 1596, :53:25, :54:58,YHLH 1596, :54:59, :57:41,AHHJ 1596, :57:42, :04:40,AHCA (a) Fragmnt of th original log data. Each lin corrsponds to a tst xcution with and nd tim (b) Log fragmnt in MXML format. A sparat audit trail ntry is cratd for th and th nd of ach tst <ProcssInstanc id="1596" dscription="tst instanc 1596">... <AuditTrailEntry> <WorkflowModlElmnt>OSWL</WorkflowModlElmnt> <EvntTyp></EvntTyp> <Timstamp> T17:33: :00</Timstamp> <Originator>unknown</Originator> </AuditTrailEntry> <AuditTrailEntry> <WorkflowModlElmnt>OSWL</WorkflowModlElmnt> <EvntTyp></EvntTyp> <Timstamp> T17:34: :00</Timstamp> <Originator>unknown</Originator> </AuditTrailEntry>... </ProcssInstanc> Fig. 3. Convrting th log into th MXML format tstd. Aftrwards th tim, th compltion tim, and th four-lttr cod for th xcutd tst ar rcordd. 4 To analyz th log data with ProM w had to convrt thm into th MXML 5 format. This was ralizd by a custom-built convrtr plug-in for th ProM import framwork 6. ProMimport facilitats log transformation tasks and provids convrtr plug-ins for a wid varity of systms to th XML format usd by ProM [15]. In th MXML format, a log is composd of procss instancs (i.., cass) and within ach instanc thr ar audit trail ntris (i.., vnts) with various attributs. Ths attributs rfr to, for xampl, data filds, timstamps, or transactional information (i.., whthr th activity was schduld, d, or d). Dpnding on th kind of information that is in th log, w may b abl to answr diffrnt qustions about th procss. Figur 3(b) dpicts th MXML log fragmnt for th tst highlightd in Figur 3(a). On can s that th and th compltion of th tst ar capturd by sparat audit trail ntris (including th corrsponding timstamps), and that th nclosing procss instanc (i.., th cas) corrsponds to th tstd machin. Not that th logging taks plac on th tst-cod lvl, and that thr is no rfrnc to th job stp in whos contxt th tst is prformd. Howvr, in addition to th log data and th job stp rfrnc squnc, ASML also providd us with an additional documnt spcifying which tst cods should b xcutd in which job stp. In this mapping, thr ar a numbr of tsts that appar in mor than on job stp (i.., ar xcutd in diffrnt phass of th tst procss). 4 Conformanc Analysis Rsults In th following, w provid a summary of th rsults from analyzing th tst procss xcution logs. (Mor dtails about th spcific procss mining tchniqus and usd ProM plug-ins can b found in our tchnical rport [23].) In Sction 5, ths rsults ar thn valuatd from an ASML prspctiv and concrt improvmnt actions ar proposd. In most domains, w usually s a larg numbr of rlativly short log tracs, i.., many procss instancs with just a fw vnts. For xampl, whn looking 4 Not that both th actual machin numbrs and th four-lttr tst cods hav bn anonymizd for confidntiality rasons. 5 Th XML schma dfinition is availabl at 6 ProMimport can b frly downloadd from

8 8 Procdings of GRCIS 2009 at procsss rlatd to patint flows, insuranc claims, traffic fins, tc., thn thr ar typically thousands of cass ach containing lss than 50 vnts. Whn w xamin th log, it bcoms clar that this tst procss has vry diffrnt charactristics, sinc thr ar just a fw cass (i.., machins) but for ach machin thr may b thousands of log vnts. In th initial data st w facd procss instancs that containd mor than log vnts (ach indicating ithr th or th compltion of a spcific tst). As mntiond arlir, th tst procss of a wafr scannr lasts for svral wks and is partly rpatd aftr th machin has bn r-assmbld at th customr, thus xplaining th hug numbr of vnts pr machin. From a largr st of machins w slctd 24 machins that fulfilld our critria: (1) th tst procss ndd to b d, (2) only includ th tst priod on th ASML (and not th customr) sit, (3) blong to th sam family (rcall that typically not mor than 50 wafr scannrs of th sam typ ar producd), and (4) not b a pilot systm (as a pilot systm is usd for dvlopmnt tsting and not for manufacturing qualification). Ths 24 cass compris log vnts in total, and th numbr of log vnts pr procss instanc (i.., th lngth of th xcutd tst squnc) rangs from 2820 to Finally, w can s that thr ar 720 diffrnt typs of audit trail ntris in th log, which corrsponds to 360 diffrnt four-lttr tst cods as ach tst is capturd by both a and vnt. Furthrmor, w ar intrstd in analyzing th job stps, i.., th tst phass that can b associatd to th rfrnc squnc. To b abl to analyz th log on th job-stp lvl, w first hav to apply crtain filtring tchniqus. Rcall that thr is no information about job stps rcordd in th log, but that w hav obtaind a documnt spcifying which tsts nd to b xcutd for ach job stp. In this mapping, thr ar 184 out of th 360 dtctd tst cods associatd to a job stp. This mans that 176 of th four-lttr cods cannot b connctd to a spcific job stp (in th rmaindr of this papr w call thm unmappd cods). Thy mainly corrspond to additional (mor spcific) tsts that ar xcutd as part of th diagnosis procss aftr a failur. At th sam tim, thr ar 49 out of th 184 mappd tst cods that ar associatd to mor than on job stp, i.., thy occur in diffrnt phass of th tst procss (in th rmaindr w call thm multipl cods). Th rst of th four-lttr cods (i.., 135 tst cods) can b unambiguously mappd onto a spcific job stp. In Figur 4 w show as an xampl how a part of th log fragmnt from Figur 3 is transfrrd to th job-stp lvl 7 using a combination of multipl log filtrs. As a first stp, w mappd ach of th unambiguous tst cods onto thir corrsponding job stp idntifir, or th multipl or unmappd catgory if this was not possibl. For xampl, Figur 4 shows that th tsts OSWL, OSSP, and AHZI ar associatd to th job stp, whil th tst DSNA cannot b mappd to any job stp (i.., unmappd ). As a nxt stp, w abstractd from all vnts that occurrd btwn th first and th last vnt blonging to th sam job stp in a row. For xampl, in Figur 4 only th bginning of th first 7 Not that, again, th actual job stp nams hav bn rplacd by simpl lttr cods for confidntiality rasons.

9 Procdings of GRCIS Job-stp lvl unmappd 2 Rptitionsto-activity Filtr unmappd unmappd 1 Rmap Filtr unmappd unmappd 0 Tst-cod lvl OSWL OSWL OSSP OSSP AHZI AHZI DSNA DSNA OSWL OSSP AHZI DSNA Fig. 4. A combination of filtring tchniqus was applid to bring th log data from th tst-cod lvl to th job-stp lvl occurrnc of a tst in job stp (i.., tst OSWL ) and th nd of th last occurrnc in job stp (i.., tst AHZI ) is rtaind. Not that using this mapping, now also idl tims within on job stp ar covrd by th ovrall jobstp duration (for xampl, th idl tim btwn th compltion of tst OSWL and th of tst OSSP ). As a rsult, only changs btwn job stps bcom visibl in th log, which w will us in th following for procss discovry on th job-stp lvl. W now want to apply procss discovry tchniqus to gain insight into th actual flow of th tst procss to find out whr r-xcutions wr oftn ncssary. Procss discovry algorithms automatically construct a procss modl basd on th bhavior that was obsrvd in th vnt log. Whil it is intrsting to visualiz dpndncis on th tst-cod lvl, w also want to analyz th procss on th job-stp lvl to compar th discovrd modl to th xisting rfrnc squnc. Th translatd rfrnc squnc is dpictd in Figur 5(a), and it rflcts th normal flow of th tst squnc if nothing gos wrong (i.., if no tst fails). W alrady know that in rality parts of th tst squnc nd to b rpatd in crtain occasions. This also bcoms visibl in th discovrd modl basd on th log filtrd for job stp xcutions (cf. Figur 4), which is dpictd in Figur 5(b). Not that th discovrd procss modl allows for considrably mor paths than th rfrnc modl. Figur 6(a), shows th framd part of th mining rsult in mor dtail, whr on can asily rcogniz th rptitiv natur of th ral (as opposd to th idal, i.., rfrnc) tst procss. Not that th numbrs nxt to th arcs show th importanc of th diffrnt paths (th lowr numbr indicats how oftn this connction was obsrvd in th log, whil th uppr numbr indicats th huristic strngth of th corrsponding connction). So far, w hav sn that it is possibl to automatically discovr modls which rprsnt th bhavior that was obsrvd in th vnt log. But how wll is th actual procss rprsntd by ths modls? And to which xtnt dos th obsrvd procss comply with th bhavior spcifid in th rfrnc modl?

10 10 Procdings of GRCIS 2009 (a) Rfrnc modl (b) Discovrd procss modl Fig. 5. Translatd rfrnc squnc and discovrd procss modl on job-stp lvl Whr in th procss do most of th dviations occur? Ths ar qustions that ar addrssd by conformanc tchniqus. In th following w us conformanc chcking [22] to analyz th conformanc of both th rfrnc modl and th discovrd modl on th job-stp lvl (cf. Figur 5) with rspct to our tst log. Nxt to visualizing th discrpancis btwn an vnt log and a givn procss modl, conformanc chcking can also b usd to masur th dgr of fitnss basd on th amount of missing and rmaining tokns during log rplay [22],

11 Procdings of GRCIS Tabl 1. Fitnss valus (f) indicating th dgr of complianc for ach of th tst instancs with rspct to both th rfrnc modl and a discovrd procss modl. Clarly, th discovrd modl fits much bttr than th rfrnc modl Machin ID Rfrnc Modl Discovrd Modl Job-stp Evnts Tst-cod Evnts 0431 f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = f = Avrag f = f = i.., it quantifis to which dgr th log tracs comply with a givn procss modl. This fitnss analysis clarly indicats that th discovrd modl is much mor rprsntativ for th obsrvd tst procss than th rfrnc modl (cf. fitnss valus in Tabl 1). Tabl 1 contains th fitnss valus for ach of th tst instancs with rspct to both th rfrnc squnc and th discovrd modl on th job-stp lvl as dpictd in Figur 5, whras possibl valus rang from 0.0 (corrsponds to th cas whr th modl and th log do not fit at all) to 1.0 (i.., modl and log fit to 100%). Furthrmor, it shows how many job stp xcutions wr containd in th filtrd log for ach machin (column bfor th last column in Tabl 1), and how many tst cod vnts wr originally rcordd for this machin (last column in Tabl 1). Finally, in th bottom row avrag valus ar givn for all th 24 machins. W can s that, although th discovrd procss modl dos not ly match th bhavior obsrvd in th log, it clarly fits much bttr than th rfrnc squnc. This is not surprising as w alrady know that in contrast to th discovrd modl th rfrnc modl dos not captur th possibl rptitions in th procss at all, but it

12 12 Procdings of GRCIS 2009 dscribs th idal flow of th procss if nothing gos wrong. So, th discovrd modl is a much bttr rprsntation of th tst procss as it took plac, which dmonstrats that procss mining can provid insight into how procsss ar rally xcutd. 5 Evaluation From ASML Prspctiv To idntify concrt improvmnt suggstions, w valuatd th prsntd procss mining rsults from an ASML prspctiv. For this, th ordr of job stps was analyzd. Th job-stp ordr is th squnc in which job stps ar xcutd in th factory. Som variation is allowd, but not too much. W invstigatd whthr according to th discovrd modl as in Figur 5 th tst procss followd th rfrnc procss (including th allowd variations). Whn w invstigatd whthr th ral procss followd th rfrnc procss, considring th allowd variations, w obtaind thr typs of rsults: (1) job stps that ar actually xcutd on a diffrnt plac in th rfrnc squnc (i.., dviations from th procss modl shown in Figur 5(a)), (2) groups of highly connctd job stps, and (3) job stps that ar not in th rfrnc squnc but in th tst log. In th following, w dscrib thm in mor dtail. (1) It appard that job stp i was positiond in 81% of th cass just aftr th zro job stp, i.., at th bginning of th discovrd procss modl, whil according to th rfrnc squnc it should b xcutd in th middl of th tst procss. Whil looking for possibl root causs for this diffrnc, w ralizd that a nwr vrsion of th rfrnc squnc was rlasd in th nd of Th main chang in th nw rfrnc squnc was that job stp i and j wr positiond just aftr th zro job stp at th bginning of th tst squnc. Th analyzd systms wr build up according to th nw squnc for job stp i. Intrstingly, job stp j was still found in th original position. If job stp j is rally to b xcutd in th bginning of th squnc, thn activ string should tak plac to align th tst xcution. Not that w also r-chckd th conformanc of th tst log with rspct to th updatd rfrnc squnc, but th fitnss valus did not chang significantly (on avrag f 0.45). (2) Two highly connctd groups of job stps ar includd in th discovrd procss modl. Th first group is dpictd in Figur 6(a), a strong connction btwn job stp f and a numbr of othr job stps:, b, g and o. Ths connctions ar bi-dirctional btwn f and th othr job stps. A rason for this ffct could b that any xcution of th job stps, b, g and o rsults in a r-xcution of job stp f. Job stp f is a rlativly short job stp which can b xcutd automatically. As a rsult, th ntir tst st is xcutd. Spcific parts of th tst st in job stp f could b fastr whn job stp f nds to b xcutd aftr job stp, b, g and o ar xcutd. In gnral, spding up job stp f is bnficial bcaus it is xcutd multipl tims in th ntir squnc. Th scond highly connctd group is cntrd around job stps r, s, t, and j. Th mind procss showd th following pattrn (s Figur 6(b)). Job

13 Procdings of GRCIS (a) Framd ara in Figur 5(b) (b) Othr group Fig. 6. Highly connctd groups of job stps, which hav bn idntifid basd on th procss mining rsults prsntd in Sction 4 stp r and t ar bi-dirctionally connctd. Job stps r, j and t ar illumination stps (i.., th wafr is xposd by light), whil job stp s is a non-illumination stp. Th root caus of a failur of job stp t is solvd by job stp r. A r-xcution of job stp r causs a r-xcution of job stp s (and possibly j ). An improvmnt proposal would b to introduc a mor thorough tst in job stp r (i.., add a similar tst to th on in job stp t ) which causs that, if th failur occurs, it alrady occurs in job stp r and can b immdiatly fixd in job stp r. This prvnts th r-xcution of job stp s (and possibly j ). (3) On of th fdback loops rvald that job stp d is xcutd, although it is not in th rfrnc squnc. Job stp d is currntly not invstigatd to b improvd to dcras th cycltim, bcaus this job stp is not supposd to b xcutd. Th procss mining rsults rvald that job stp d is xcutd as part of a rcovry plan. Job stp d could b furthr invstigatd for cycl tim rduction. Th abov analysis illustrats that procss mining can b applid to chck th conformanc of procsss, i.., dviations can b dtctd and analyzd. 6 Conclusion Using a tst procss in ASML, w hav illustratd th applicability of ProM s conformanc chckr. Th cas study clarly shows that, givn an vnt log and a procss modl as input, conformanc chcking can b usd to dtct dviations. Th svrity of ths dviations can b qualifid and possibl causs can b analyzd. Hnc, conformanc chcking is a usful tool in assssing rgulatory complianc. Th cas study is a bit a-typical, i.., rgulatory complianc is oftn associatd with rgulations such as th Sarbans-Oxly Act, Basl II, and HIPAA. Ths rgulations focus on banks, insuranc companis, govrnmntal agncis, hospitals, tc. Howvr, ProM s conformanc chckr is gnric and applicabl to any notation that can b mappd onto Ptri nts. Morovr, th cas study

14 14 Procdings of GRCIS 2009 within ASML illustrats th trnd that mor and mor dvics ar connctd to th intrnt. Anothr xampl is th CUSTOMrCARE Rmot Srvics Ntwork of Philips Halthcar (PH). This is a worldwid intrnt-basd privat ntwork that links PH quipmnt to rmot srvic cntrs. Any vnt that occurs within an X-ray machin (.g., moving th tabl, stting th dflctor, tc.) is rcordd and can b analyzd [16]. Th logging capabilitis of th machins of PH illustrat th incrasing availability of vnt data. Th omniprsnc of such dtaild logging will hav dramatic ffcts on complianc. Whil today many procsss ar not auditabl bcaus vital information is missing, it is clar that much mor audit data will b availabl in th nar futur. Acknowldgmnts. This rsarch is supportd by th Tchnology Foundation STW, EIT, and th Tangram and IOP programs of th Dutch Ministry of Economic Affairs. Furthrmor, w want to thank ASML for thir coopration. Rfrncs 1. W.M.P. van dr Aalst, B.F. van Dongn, C.W. Günthr, R.S. Mans, A.K. Alvs d Mdiros, A. Rozinat, V. Rubin, M. Song, H.M.W. Vrbk, and A.J.M.M. Wijtrs. ProM 4.0: Comprhnsiv Support for Ral Procss Analysis. In J. Klijn and A. Yakovlv, ditors, Procdings of th ICATPN 2007, volum 4546 of Lctur Nots in Computr Scinc, pags Springr-Vrlag, Brlin, W.M.P. van dr Aalst, B.F. van Dongn, J. Hrbst, L. Marustr, G. Schimm, and A.J.M.M. Wijtrs. Workflow Mining: A Survy of Issus and Approachs. Data and Knowldg Enginring, 47(2): , W.M.P. van dr Aalst, A.J.M.M. Wijtrs, and L. Marustr. Workflow Mining: Discovring Procss Modls from Evnt Logs. IEEE Transactions on Knowldg and Data Enginring, 16(9): , R. Agrawal, D. Gunopulos, and F. Lymann. Mining Procss Modls from Workflow Logs. In Sixth Intrnational Confrnc on Extnding Databas Tchnology, pags , R. Agrawal, C.M. Johnson, J. Kirnan, and F. Lymann. Taming complianc with sarbans-oxly intrnal controls using databas tchnology. In Procdings of th 22nd Intrnational Confrnc on Data Enginring (ICDE 2006), pag 92. IEEE Computr Socity, R. Boumn, I.S.M. d Jong, J.M.G. Mstrom, J.M. van d Mortl-Fronczak, and J.E. Rooda. Intgration and Tst Squncing for Complx Systms. IEEE Transactions on Systms, Man, and Cybrntics, Part A, 39(1): , R. Boumn, I.S.M. d Jong, J.W.H. Vrmunt, J.M. van d Mortl-Fronczak, and J.E. Rooda. Tst Squncing in Complx Manufacturing Systms. IEEE Transactions on Systms, Man, and Cybrntics, Part A, 38(1):25 37, J.E. Cook, C. H, and C. Ma. Masuring Bhavioral Corrspondnc to a Timd Concurrnt Modl. In Procdings of th 2001 Intrnational Confrnc on Softwar Mainnanc, pags , J.E. Cook and A.L. Wolf. Softwar Procss Validation: Quantitativly Masuring th Corrspondnc of a Procss to a Modl. ACM Transactions on Softwar Enginring and Mthodology, 8(2): , 1999.

15 Procdings of GRCIS A. Datta. Automating th Discovry of As-Is Businss Procss Modls: Probabilistic and Algorithmic Approachs. Information Systms Rsarch, 9(3): , A. Ghos and G. Koliadis. Auditing Businss Procss Complianc. In B.J. Kramr, K.J. Lin, and P. Narasimhan, ditors, Fifth Intrnational Confrnc on Srvic- Orintd Computing (ICSOC 2007), volum 4749 of Lctur Nots in Computr Scinc, pags Springr-Vrlag, Brlin, G. Govrnatori, J. Hoffmann, S. Sadiq, and I. Wbr. Dtcting Rgulatory Complianc for Businss Procss Modls through Smantic Annotations. In 4th Intrnational Workshop on Businss Procss Dsign, G. Govrnatori, Z. Milosvic, and S. Sadiq. Complianc Chcking Btwn Businss Procsss and Businss Contracts. In 10th Intrnational Entrpris Distributd Objct Computing Confrnc (EDOC 2006), pags IEEE Computing Socity, D. Grigori, F. Casati, M. Castllanos, U. Dayal, M. Sayal, and M.C. Shan. Businss procss intllignc. Computrs in Industry, 53(3): , C.W. Günthr and W.M.P. van dr Aalst. A Gnric Import Framwork for Procss Evnt Logs. In J. Edr and S. Dustdar, ditors, Businss Procss Managmnt Workshops, Workshop on Businss Procss Intllignc (BPI 2006), volum 4103 of Lctur Nots in Computr Scinc, pags Springr-Vrlag, Brlin, C.W. Günthr, A. Rozinat, W.M.P. van dr Aalst, and K. van Udn. Monitoring Dployd Application Usag with Procss Mining. BPM Cntr Rport BPM-08-11, BPMcntr.org, H. Hss. Monitoring, Analyzing and Optimizing Corporat Prformanc: Stat of th Art and Currnt Trnds. In Agility by ARIS Businss Procss Managmnt, pags Springr-Vrlag, Brlin, IDS Schr. ARIS Procss Prformanc Managr (ARIS PPM): Masur, Analyz and Optimiz Your Businss Procss Prformanc (whitpapr). IDS Schr, Saarbruckn, Gmany, Y. Liu, S. Mullr, and K. Xu. A Static Complianc-Chcking Framwork for Businss Procss Modls. IBM Systms Journal, 46(2): , A.K. Alvs d Mdiros, A.J.M.M. Wijtrs, and W.M.P. van dr Aalst. Gntic Procss Mining: An Exprimntal Evaluation. Data Mining and Knowldg Discovry, 14(2): , M. zur Mühln and M. Rosmann. Workflow-basd Procss Monitoring and Controlling - Tchnical and Organizational Issus. In R. Spragu, ditor, Procdings of th 33rd Hawaii Intrnational Confrnc on Systm Scinc (HICSS-33), pags IEEE Computr Socity Prss, Los Alamitos, California, A. Rozinat and W.M.P. van dr Aalst. Conformanc Chcking of Procsss Basd on Monitoring Ral Bhavior. Information Systms, 33(1):64 95, A. Rozinat, I.S.M. d Jong, C.W. Günthr, and W.M.P. van dr Aalst. Procss Mining of Tst Procsss: A Cas Study. BETA Working Papr Sris, WP 220, Eindhovn Univrsity of Tchnology, Eindhovn, A. Rozinat, I.S.M. d Jong, C.W. Günthr, and W.M.P. van dr Aalst. Procss Mining Applid to th Tst Procss of Wafr Stpprs in ASML. Accptd for publication in IEEE Transactions on Systms, Man, and Cybrntics Part C, M. Sayal, F. Casati, U. Dayal, and M.C. Shan. Businss Procss Cockpit. In Procdings of 28th Intrnational Confrnc on Vry Larg Data Bass (VLDB 02), pags Morgan Kaufmann, 2002.