Evaluation of Literature-based Discovery Systems

Transcription

1 Evaluaton of Lterature-based Dscovery Systems Melha Yetsgen-Yldz 1 and Wanda Pratt 1,2 1 The Informaton School, Unversty of Washngton, Seattle, USA. 2 Bomedcal and Health Informatcs, School of Medcne, Unversty of Washngton, Seattle, USA. {melhay,wpratt}@u.washngton.edu Abstract. Evaluatng dscovery systems s a fundamentally challengng task because f they are successful, by defnton they are capturng new knowledge that has yet to be proven useful. To overcome ths dffculty, many researchers n lterature-based dscovery (LBD) replcated Swanson's dscoveres to evaluate the performance of ther systems. They reported overall success f one of the dscoveres generated by ther system was the same as Swanson's dscovery. Ths type of evaluaton s powerful yet ncomplete because t does not nform us about the qualty of the rest of the dscoveres dentfed by the system nor does t test the generalzablty of the results. Recently, alternatve evaluaton methods have been desgned to provde more nformaton on the overall performance of the systems. The purpose of ths chapter s to revew and analyze the current evaluaton methods for LBD systems and to dscuss potental ways to use these evaluaton methods for comparng performance of dfferent systems, rather than reportng the performance of only one system. We wll also summarze the current approaches used to evaluate the graphcal user nterfaces of LBD systems. 1 Introducton Evaluaton plays an mportant role n the development of new felds such as lteraturebased dscovery (LBD). Evaluaton encourages scentfc progress by supportng a systematc comparson of dfferent technques appled to a common problem and allowng researchers to learn from each other s successes and falures. In ths chapter, we wll gve an overvew of the current state of evaluaton n lterature-based dscovery research and dscuss potental ways for future evaluatons. 2 Evaluaton Metrcs When developng an LBD system, t s crtcal to know how relable the results are lkely to be. Measurng the relablty of a predcton system requres two man components: a gold standard and an evaluaton metrc to measure the system s performance wth respect to the gold standard. For a gven startng term, whch

2 2 Melha Yetsgen-Yldz and Wanda Pratt Swanson called C-Term, a typcal LBD system produces two sets of terms; lnkng terms and target terms. The lnkng terms, whch Swanson called B-Terms, drectly connect a gven startng term to the target terms, whch Swanson called A-Terms. The gold standards used to evaluate those two sets of terms are dfferent from each other, and the gold standard creaton methods depend on whch of the evaluaton methods lsted n Secton 3 s used. We wll descrbe how the gold standards for lnkng/target terms are created for certan types of evaluaton methods n Secton 3. For now, we wll defne the gold standards for lnkng/target terms as the two sets of terms that are known to be drectly/ndrectly connected to a gven startng term. In ths secton, we wll summarze the metrcs used to measure the performance of LBD systems. 2.1 Informaton Retreval Metrcs The man purpose of evaluaton n nformaton retreval research (IR) s to measure IR systems performance n returnng the relevant documents and n not returnng the non-relevant documents to user queres. In IR evaluaton, the gold standard s the set of relevant documents and two most popular IR metrcs used to measure system performance are precson and recall [1]. For a gven query and an IR system, precson can be defned as the proporton of relevant documents n the set of documents returned by the system and recall can be defned as the proporton of the relevant documents retreved by the system from the gold standard. In contrast to IR systems, LBD systems return terms nstead of documents. Thus, precson and recall are manly used to measure the effectveness of an LBD system n returnng lnkng and target terms for a gven startng term, rather than the effectveness of an IR system n returnng documents for a gven query. Precson and recall for the LBD system evaluaton are calculated wth the followng formulas: T G Precson: P = (1) T Recall: R T G = (2) G where T s the set of lnkng/target terms generated by the LBD system for the startng term, and G s the set of terms n the lnkng/target term gold standard that the LBD system created for the startng term. As wth IR system evaluaton, one challenge n nterpretng precson and recall s that there s a trade-off between the two metrcs. Usually a system that ams to acheve hgh precson wll result n low recall and vce versa. To solve ths problem, some nformaton retreval researchers nvented a new measure called F-Measure whch s a combned verson of precson and recall. F-Measure s calculated wth the followng formula: 2 (1 + β ) R P F-Measure: F = (3) 2 ( β P) + R

3 Evaluaton of Lterature-based Dscovery Systems 3 where R s the recall, P s the precson, and β s the relatve value of the precson. The most commonly used case β = 1 assgns equal emphass on precson and recall, whereas a lower value assgns a hgher emphass on precson and a hgher value assgns a hgher emphass on recall. Another common method to combne precson and recall s to draw a precsonrecall curve. In ths curve, the x-axs corresponds to recall and the y-axs corresponds to precson. Because of the trade-off between precson and recall, precson-recall graphs usually have a concave shape. Tryng to ncrease recall typcally ntroduces more false postves (target terms that are not n the gold standard), and thereby reduces precson. Tryng to ncrease precson typcally reduces recall by decreasng the number of true postves (target terms that are n the gold standard). An deal goal of a predcton system s to ncrease both precson and recall by makng mprovements to the system. In other words, the entre curve must move up and out to the rght so that both recall and precson are hgher at every pont along the curve. The most common use of precson-recall curves s for system comparsons. 2.2 Recever Operatng Characterstcs (ROC) Curve Recever Operatng Characterstcs (ROC) curve provdes a graphcal representaton of the relatonshp between the true postve and false postve rate of a predcton system [2]. These curves are used frequently n comparng the effectveness of dfferent medcal dagnostc tests. The y-axs corresponds to the senstvty of the system. Senstvty measures the performance of the system n predctng the true postves. The x-axs corresponds to the specfcty (expressed as 1-specfcty n the graph). Specfcty represents the ablty of the system n dentfyng true negatves. The senstvty and the specfcty of a LBD system can be calculated as: TP Senstvty: Y = (4) TP + FN 1- Specfcty: X TN = 1 (5) TN + FP where for the startng term ; TP s the number of true postves (the target terms that are n the gold standard), FN s the number of false negatves (the gold standard terms that are not dentfed as target terms), FP s the number of false postves (the target terms that are not n the gold standard), and TN s the number of true negatves (the terms that are both not selected as target terms and not n the gold standard). The ROC curves show the performance as a trade off between specfcty and senstvty of the predcton system. The area under the ROC s a convenent way of comparng dfferent predcton systems. A random system has an area of 0.5, whle and deal one has an area of 1.

4 4 Melha Yetsgen-Yldz and Wanda Pratt 2.3 Probablstc Approaches Because the purpose of LBD systems s to predct novel connectons between medcal terms, t s also mportant to compare ther predcton performance wth that of pure random predcton. One way to accomplsh ths objectve s to calculate the probablty of randomly achevng the performance of a gven LBD system. Ths probablty can be modeled wth hypergeometrc dstrbuton. Suppose for a gven startng term, an LBD system returns k target terms where of the target terms that are n the gold standard, there are n terms n the gold standard and there are m terms n the search space of the system. The probablty of havng gold standard terms n randomly selected k target terms s calculated wth the followng formula: n m n k p ( x = ) = (6) m k If the value of p s close to zero, achevng the performance of the LBD system by randomly selectng the target terms s hghly unlkely. If the value of p s close to 1, the predcton of mechansm of the LBD system needs to be mproved because random selecton of the terms gves almost the same performance. 3 Current Evaluaton Approaches Evaluatng the performance of LBD systems s a fundamentally challengng task because f these systems are successful, by defnton, they are capturng new knowledge that has yet to be proven useful. After a detaled analyss of the exstng lterature on LBD systems, we dentfed the followng four dfferent approaches used to evaluate LBD systems; replcatng Swanson s dscoveres, usng statstcal evaluaton approaches, ncorporatng expert knowledge, and publshng n the medcal doman. In ths secton, we wll explan each evaluaton approach n detal and dscuss ther advantages and dsadvantages. 3.1 Replcatng Swanson s Experments Even though the LBD systems are desgned to produce new knowledge, measurng ther performance by replcatng the hstorcal dscoveres has been seen an effectve evaluaton approach by many LBD researchers. Swanson and Smalheser publshed several dfferent hypotheses about causally connected medcal terms n the bomedcal doman ncludng Mgrane Magnesum [3], Raynaud s Dsease - Fsh Ol [4], Alzhemer s Dsease Estrogen [5], Alzhemer s Dsease Indomethacn [6], Somatomedn C Argnne [7], and Schzophrena Calcum Independent Phospholpase 2 A [8]. Ther dscoveres have become gold standards for evaluaton, and LBD researchers have measured the performance of ther dscovery systems by

5 Evaluaton of Lterature-based Dscovery Systems 5 replcatng Swanson s dscoveres usng the lterature publshed before the orgnal dscovery dates. They have run ther systems wth Swanson s startng terms on the lterature publshed pror to the dscovery dates and reported overall success f one of the correlatons generated by ther systems matched Swanson s dscovery. Several researchers have used ths strategy to evaluate the lnkng terms generated by ther systems. Lndsay and Gordon [9] developed a process that followed the Swanson s dscovery approach. They evaluated the performance of ther process, n terms of precson and recall, for generatng the lnkng terms, where Swanson s dentfed lnkng terms for Mgrane-Magnesum example served as the gold standard. Gordon and Dumas appled latent semantc ndexng to Swanson s dscovery process [10]. They demonstrated the performance of ther approach by replcatng Swanson s Raynaud s Dsease and Fsh Ol dscovery. Blake and Pratt appled a knowledge-based approach to dentfy and prune potental lnkng terms [11]. They replcated Swanson s Mgrane-Magnesum example to evaluate ther approach. However, all of these researchers focused on evaluatng the lnkng terms by usng Swanson s lnkng terms as the gold standard, and none pursued or evaluated how easy t would be dentfy the novel target term (e.g., magnesum), whch s the man goal of LBD systems.. Weeber et. al. also based ther work on Swanson s approach [12]. They evaluated ther lterature-based dscovery tool DAD by smulatng Swanson s Raynaud s Dsease-Fsh Ol and Mgrane-Magnesum examples. Ther system supported both open and closed dscovery approaches. In the open dscovery approach, DAD frst dentfed the lnkng terms that are drectly connected to the startng terms, Raynaud s Dsease and Mgrane, and then dentfed the target terms that are connected to the lnkng terms dentfed n the frst step. They reported whch of the Swanson s lnkng terms DAD could dentfy and the ranks of Fsh Ol and Magnesum n the fnal lsts of target terms. In the closed dscovery approach, they analyzed the startng term lterature and the target term lterature separately and dentfed the overlappng terms. They compared those terms wth Swanson s lnkng terms and reported the results. The most extensve evaluaton of ths type was done by Srnvasan [13]. She developed a lterature based dscovery system called Manjal. As Weeber et. al. s system, Manjal supports both open and closed dscovery approaches. To evaluate her system, Srnvasan successfully replcated fve of Swanson s dscoveres ncludng Raynaud s Dsease-Fsh Ol, Mgrane-Magnesum, Alzhemer s Dsease- Indomethacn, Somatomedn C-Argnne, and Schzophrena-Calcum Independent Phospholpase A2. For each dscovery, she reported the rank of the desred target term n the lst of target terms generated by Manjal wth the open dscovery approach. She also reported the ranks of the desred lnkng terms dentfed by Manjal wth the closed dscovery approach. Most recently, Hu et. al. developed a prototype system called Bo-SbKDS based on Swanson s dscovery approach [14]. They replcated Swanson s Mgrane- Magnesum and Raynaud Dsease-Fsh Ol dscoveres for evaluaton purposes. He used Mgrane and Raynaud s Dsease as startng terms. They reported whch of Swanson s lnkng terms ther system could dentfy as lnkng terms and the ranks of Magnesum and Fsh Ol n the fnal lsts of target terms generated by ther system.

6 6 Melha Yetsgen-Yldz and Wanda Pratt In prevous research, we also replcated Swanson s Mgrane-Magnesum dscovery to evaluate the capabltes of our system LtLnker [15]. As other researchers, we compared our lnkng terms wth Swanson s lnkng terms and reported the rank of Magnesum n the fnal lst of target terms. The man advantage of ths type of evaluaton s the ease of desgnng t. In hs papers, Swanson descrbed each of hs dscoveres n great detal. The researchers use the nformaton provded n those papers as a gude n desgnng ther evaluatons. For each dscovery, the publcaton date of the correspondng paper serves as the orgnal dscovery date and the lst of medcal terms he used as lnks between hs startng term and target term serves as a lnkng term gold standard. Although all the researchers mentoned n ths secton have successfully replcated Swanson s dscoveres, ths type of evaluaton s not complete because t does not nform us about the qualty of the rest of the target terms dentfed by ther systems. Dependng on the approaches used to select the correlated terms, a lterature-based dscovery system mght return hundreds or even thousands of terms as the target terms for a gven startng term. Evaluatng the whole system on only one of those target terms does not guarantee that the rest of the target terms also provde nformaton wth smlar qualty. As wth nformaton retreval systems, an LBD system that returns a sngle helpful target term n a sea of unhelpful target terms s unlkely to be useful. Another dsadvantage of ths approach s that the researchers are lmted n ther evaluatons to the small number of dscoveres publshed by Swanson. Hs dscoveres mostly focused on dseases and ther potental new treatments. Nevertheless, LBD tools can be used for varous other tasks, such as dentfyng novel proten-proten nteractons. Because the researchers know exactly what they are seekng as the desred target and lnkng terms n ths lmted set of dscoveres, they can tune the parameters of ther systems to be able to dentfy those terms. Such an approach mght result n systems that perform well for the specfc example cases but not well for other cases. In addton, comparng the performance of dfferent systems s one of the man objectves of system evaluaton. However, replcatng Swanson s dscoveres does not allow detaled comparsons between dfferent LBD systems. Ths evaluaton method allows the researchers to say a system A s better than another system B f A smulates a selected dscovery but B does not. However, f both A and B successfully smulate the gven dscovery successfully, t becomes mpossble to determne whch system s superor to the other. 3.2 Usng Statstcal Evaluaton Methods To overcome the drawbacks of the prevous approach, some researchers have appled statstcal evaluaton methods to measure the overall performance of lterature-based dscovery systems for multple target terms. As an example, Hrstovsk et.al. performed a statstcal evaluaton of ther system, BITOLA [16]. The purpose of ther evaluaton was to see how many of the potental dscoveres made by ther system at a specfed pont n tme become realzed at a later tme. To accomplsh ths goal, they ran ther system for the startng term Multple Secleross on the set of documents

7 Evaluaton of Lterature-based Dscovery Systems 7 publshed between 1990 and They checked the exstence of the proposed dscoveres n the set of documents publshed between 1996 and 1999 and calculated precson and recall. They used a very lmted porton of the medcal lterature and reported the performance statstcs of ther system wthout comparng t to those of other systems. To evaluate our system LtLnker, we used a smlar but more extensve approach than Hrstovsk et.al. s approach; ths approach enabled us to evaluate all correlatons LtLnker generated. In our evaluaton, for a gven startng term, we measured whether LtLnker leads us to new dscoveres n the more recently publshed medcal lterature. To accomplsh ths goal, we dvded MEDLINE nto two sets: (1) a baselne set ncludng only publcatons before a selected cut-off date, and (2) a test set ncludng only publcatons between the cut-off date and another later date. We ran LtLnker on the baselne set and checked the generated connectons n the test set. As an evaluaton example, n [17], we ran LtLnker for the startng terms; Alzhemer Dsease, Mgrane, and Schzophrena on a baselne set, whch ncluded only documents publshed before January 1, 2004 (cut-off date). We lmted the lnkng terms and the target terms to only those terms n a semantc group lsted n Table 1 because the goal of our experments was to fnd novel connectons between the selected dseases and chemcals, drugs, genes, or molecular sequences. We checked the exstence of target terms generated by LtLnker n the test set that was composed of artcles publshed between January 1, 2004 and September 30, 2005 (21 months). Table 1. Semantc Groups selected for our experments Lnkng Term Selecton Chemcals & Drugs Dsorders Genes & Molecular Sequence Physology Anatomy Target Term Selecton Chemcals & Drugs Genes & Molecular Sequence To calculate precson and recall, for each startng term, we frst retreved the terms that co-occurred wth the startng term n the test set but dd not co-occur wth the startng term n the baselne set. Then, we fltered the retreved lst of terms by usng the semantc groups that we used for target term selectons to fnd the ones that were chemcals, drugs, genes, or molecular sequences. We assumed that the terms n the remanng lst would be new potental dsease to gene or dsease to drug treatment dscoveres and used them as the target term gold standard for our precson and recall calculatons. In our current research, we used our evaluaton approach to compare two dfferent methods for dentfyng lnkng or target terms based on a startng term, Z-Score [17] and MIM [18]. To accomplsh ths task, we frst mplemented the methods wthn our LtLnker framework. In our experments, for each method, we ran LtLnker for 10

8 8 Melha Yetsgen-Yldz and Wanda Pratt randomly selected dsease names on a baselne set, whch ncludes only documents publshed before January 1, We created a target term gold standard for each dsease from the test set documents publshed between January 1, 2004 and July, 31, 2006 (31 months). We calculated precson and recall of both methods for each dsease and ran statstcal sgnfcance tests to measure the sgnfcance of the performance dfferences. We also used precson-recall graphs to compare dfferent correlaton methods. To draw precson-recall graphs, we used the ranked lst of target terms generated by the two methods. We examned these lsts of target terms startng from the top and selected ntervals to calculate precson and recall wth the formulas (1) and (2). Because we had 10 dfferent startng terms, to combne the results from each experment, we calculated the average precson and recall for each nterval. We also compared the predcton performances of both methods wth that of pure random predcton wth hypergeometrc dstrbuton as descrbed n Secton 2.3. The man advantages of ths type of evaluaton are that the evaluaton s fully automated, can be repeated for multple startng terms, and enables comparson among dfferent systems. On the other hand, ts man drawback s that the calculated precson for target terms s the lower bound. The target term gold standard only ncludes the new correlatons that are publshed between the cut-off date and the date of the experment. It cannot nclude the correlatons that wll appear n the future. As a result, some of the target terms dentfed by the LBD system mght become legtmate dscoveres n the future but are consdered ncorrect target terms now. Another dsadvantage s that ths approach only evaluates the target terms wthout provdng any nformaton about the lnkng terms. 3.3 Incorporatng Expert Opnon As an alternatve to the prevous approaches, some researchers ncorporated medcal expert knowledge to the evaluaton process of ther LBD systems. Weeber et. al., used ther dscovery system to nvestgate new potental uses for drug thaldomde wth Swanson s open dscovery approach [19]. One of the researchers nvolved n ths study was a medcal researcher wth a background on pharmacology and mmunology. For the startng term thaldomde, ther system generated a lst of lnkng terms that were constraned to be mmunologc factors. They manually selected the promsng lnkng terms wth the nvolvement of the medcal researcher. For the selected lnkng terms, ther system generated a lst of target terms that were constraned to be dsease or syndrome names. The medcal researcher manually assessed each of the selected dseases. In the assessment process, they tred to fnd addtonal bblographc and other evdence for the lnkng terms between the thaldomde and the dseases dentfed as target terms. To accomplsh ths goal, for each dsease, they frst extracted the lst of lnkng terms that connect the dsease to thaldomde. Next, they extracted the sentences that ncluded thaldomde and the extracted lnkng terms and the sentences that ncluded the lnkng terms and the dsease. They provded those sentences to the medcal expert for assessment. Based on the assessment, they compled a lst of four dseases; chronc hepatts C,

9 Evaluaton of Lterature-based Dscovery Systems 9 myasthena gravs, helcobacter pylor nduced gastrts, acute pancreatts for whch the researchers hypotheszed that thaldomde could be an effectve treatment. Srnvasan and Lbbus evaluated ther system Manjal by usng a sem-automated approach wth experts. In ther experment, they used turmerc, a wdely used spce n Asa, as ther startng term. The am of ther experment was to dentfy dseases where turmerc could be useful n the treatng them. They ran Manjal for the startng term turmerc, and, wth the selected thresholds, Manjal dentfed 26 terms as the lnkng terms, L 1. To evaluate the lnkng terms n L 1, a medcal researcher dentfed a second set of lnkng terms, L 2, after readng the documents about turmerc. There were 27 terms n L 2. They used ths manually created lst as the lnkng term gold standard. They compared L 1 wth L2 and calculated recall and precson wth the followng formulas: L1 L2 Precson: P = (7) L L1 L2 Recall: R = (8) L2 Manjal generated two sets of target terms; one from the automatcally generated lnkng terms and one from the manually selected lnkng terms. They used the second set as the target term gold standard to evaluate the frst set and reported precson and recall. In addton to reportng precson and recall, they dd a detaled ctaton analyss and descrbed the potental use of turmerc n the treatment of retnal dseases, Crohn s dsease, and spnal cord njures. In contrast to the statstcal approach descrbed n the prevous secton, the advantage of Srnvasan and Lbbus s approach s that t allows us to evaluate the lnkng terms n addton to the target terms. However, the evaluaton hghly depends on the subjectve decson of the medcal researcher n decdng whch terms are correlated wth the startng term. Ths decson s crucal because t also drectly effects the selecton of the terms n the target term gold standard. It s also unclear whether the gold standard set of target terms reflects a true gold standard because no checkng has been done on those target terms. Wren et.al. also ncorporated medcal expert knowledge nto the evaluaton process [20]. The researchers who contrbuted to ths study had a medcal background. They ran ther lterature-based dscovery approach for the startng term cardac hypertrophy and dentfed a total of 2102 lnkng terms and target terms. To evaluate ther approach, they performed laboratory tests for the 3 rd ranked target term, chlorpromazne. Chlorpromazne s a chemcal that s used as an ant-psychotc and ant-emetc drug. In ther lab experments, they looked for an assocaton between chlorpromazne and cardac hypertrophy. They gave 20mg/kg/day soproterenol by osmotc mnpump to two groups of mce, wth one group addtonally recevng 10mg/kg/day chlorpromazne. Ther results showed that the amount of cardac hypertrophy was sgnfcantly reduced n the soproterenol plus chlorpromazne treated mce n comparson to the control group only gven soproterenol. They 1

10 10 Melha Yetsgen-Yldz and Wanda Pratt reported that chlorpromazne could reduce cardac hypertrophy by showng ther expermental results wth mce as evdence. Ther work s an excellent example of how lterature-based dscovery tools can be ntegrated to medcal researcher s real-lfe research actvtes. The man advantage of ths type of evaluaton s the nvolvement of the medcal researchers, who are the real users of the LBD systems nto the evaluaton process. To dentfy what medcal researchers fnd nterestng or not nterestng could nform LBD system desgners whle they upgrade the algorthms or the other approaches they use n the dscovery process. The downsde s the hgh cost of evaluaton. Weeber et. al. reported that ther manual effort whle evaluatng the output of ther system conssted of several one hour sessons durng a two-week perod. Such an evaluaton s also hard to quantfy, and thus hard to use to compare dfferent LBD systems. Because the am of LBD tools s to dentfy novel correlatons, dsagreements on the nterestngness of the correlatons could arse f multple medcal researchers are nvolved n the evaluaton process. 3.4 Publshng n the Medcal Doman Another approach that s used to evaluate LBD systems s publshng the dscoveres n medcal journals or presentng them n the medcal doman. Ths evaluaton approach s a very powerful yet a very challengng one. Publshng n the medcal doman requres the flexblty to wrte for the medcal audence, but the overall beneft s clear: valdaton of work, mpact on the scence, external vsblty for LBD research, and the chance to gan new collaborators. Ths type of evaluaton s not commonly used n LBD research. Among all LBD researchers, Swanson s the only researcher who could publsh hs dscoveres n the medcal journals. In addton to Swanson s personal nterest n medcne, hs close collaboraton wth Smalheser who s a medcal doctor and neuroscentst, resulted n varous publcatons [3-8, 21]. 4 User Interface Evaluaton The success of an LBD system n facltatng new dscoveres depends on ts nterface s ablty to nform and engage ts users as they attempt to nterpret and evaluate the proposed connectons. The amount of data produced by an LBD system s usually mmense. As an example, when LtLnker replcated Swanson s Mgrane- Magnesum dscovery, t processed over 4 mllon documents. It generated 349 lnkng terms and 545 target terms wth 57,622 possble startng term-lnkng term and lnkng term-target term combnatons. To be able to handle the amount and complexty of the output data, one of the prmary objectves of an LBD system nterface must be to promote user comprehenson of numerous complex relatonshps among the terms nvolved n each proposed connecton n an effectve way. The nterface must also provde flexble navgaton and a level of detal approprate to the scope of each vew wthout obscurng data necessarly for evaluaton purposes. And most mportantly, the nterface should help researchers ncorporate the LBD system s results nto ther own research dscovery process. To accomplsh those objectves

11 Evaluaton of Lterature-based Dscovery Systems 11 requres the nvolvement of real users nto the nterface desgn process. One way to nvolve users s by conductng usablty evaluatons and changng the nterface desgn accordng to the feedback collected from the partcpants of the evaluaton. We desgned a web-based graphcal nterface for LtLnker 1. Our am n developng an nterface was to allow researchers to carefully assess the potental connectons generated by LtLnker. We frst developed a prototype nterface and conducted a usablty evaluaton wth ten partcpants, ncludng nne graduate students and one faculty member [22]. The evaluaton conssted of three parts: a general ntroducton, a task-based questonnare, and an ntervew. The partcpants used LtLnker wth Mgrane as the startng term, to complete a task-based questonnare. The tasks were desgned to evaluate each partcpant s ablty to fnd specfc data, to navgate the nterface, and to compare the strengths of connectons. Partcpants were asked to talk aloud and as they completed the tasks. The ntervewer observed wthout answerng questons and noted any dffcultes the partcpants experenced. After partcpants completed the questonnare, we ntervewed them to dscover aspects of the nterface that were confusng or were partcularly helpful. We dentfed many desgn problems durng ths usablty evaluaton and modfed our nterface to ncrease ts usablty. Smlarly, Smalheser et. al. evaluated ther LBD system, Arrowsmth as part of a fve year neuroscence project at Unversty of Illnos-Chcago [23]. The goal of ther evaluaton study ncluded makng scentfc dscoveres, publshng papers, and dentfyng new research drectons. In contrast to our study, they dd not recrut human subjects or study ther behavor on standardzed tasks. Rather, the medcal researchers who partcpated n the study chose the search topcs and observed the outcomes. Each partcpant was gven an electronc notebook to record opportuntes for conductng Arrowsmth searches, whether they arose from laboratory experments, from attendng conferences, or from dscussons wth other researchers, and to record the detals of completed Arrowsmth searches. Partcpants sent the notebook entres va e-mal to the researchers and the researchers called the partcpants every week to montor the course of ther scentfc work, to learn more about the completed searches, to receve suggestons for mprovng the nterface, and to document the follow-up of completed searches. Based on the nput they receved from the partcpants, they updated the Arrowsmth nterface. They also focused on nformaton seekng needs and strateges of medcal researchers as they formulate new hypotheses. 5 Concluson LBD systems have great promse for mprovng medcal researchers effcency whle they seek nformaton n the vast amount of lterature avalable to them. Although many onlne LBD systems are avalable, they are not n routne use. For a wder usage of LBD systems, effectve evaluaton s essental. Evaluaton wll not only help to dentfy whch algorthmc approaches work best for LBD, but also provde 1 Avalable at:

12 12 Melha Yetsgen-Yldz and Wanda Pratt nformaton about how dscovery systems can best enhance the real-lfe work processes of medcal researchers. In ths chapter, we summarzed the current evaluaton approaches used to evaluate LBD systems and ther nterfaces, but more research on evaluaton methods that standardze system comparsons and explore user behavor s needed. 6 Acknowledgements The Natonal Scence Foundaton, award #IIS , funded ths work. 7 References 1. Baeza-Yates, R., and Rbero-Neto, B., Modern Informaton Retreval. 1999: ACM Press, Addson-Wesley. 2. Bradley, A.P., The Use of the Area Under the ROC Curve n the Evaluaton of Machne Learnng Algorthms. Pattern Recognton, (7): p Swanson, D.R., Mgrane and Magnesum: Eleven Neglected Connectons. Perspect. Bol. Med., : p Swanson, D.R., Fsh Ol, Raynaud's Syndrome, and Undscovered Publc Knowledge. Perspect. Bol. Med., (1): p Smalheser, N.R., and Swanson, D., Lnkng Estrogen to Alzhemer's Dsease: An Informatcs Approach. Neurology, (3): p Smalheser, N.R., and Swanson, D., Indomethacn and Alzhemer's Dsease. Neurology, (2): p Swanson, D.R., Somatomedn C and Argnne: Implct Connectons between Mutually Isolated Lteratures. Perspectves n Bology and Medcne, (2): p Smalheser, N.R., and Swanson, D., Calcum-Independent Phospholpase A2 and Schzophrena. Arch Gen Psychatry, : p Lndsay, R.K., and Gordon, M.D., Lterature based dscovery by lexcal statstcs. Journal of Amercan Socety for Informaton Scence, (8): p Gordon, M.D., and Dumas, S., Usng latent semantc ndexng for lterature based dscovery. Journal of Amercan Socety for Informaton Scence, (8): p Blake, C., and Pratt, W. Automatcally Identfyng Canddate Treatments from Exstng Medcal Lterature.. n Proceedngs of AAAI Sprng Symposum on Mnng Answers from Texts and Knowledge Bases Calforna. 12. Weeber, M., Klen, H., and de Jong - van den Berg, L.T.W., Usng Concepts n Lterature Based Dscovery: Smulatng Swanson's Raynaud-Fsh Ol and Mgrane-Magnesum Examples. Journal of Amercan Socety for Informaton Scence, (7): p Srnvasan, P., Generatng Hypotheses from MEDLINE. Journal of Amercan Socety for Informaton Scence, (5): p Hu, X., L, G., Yoo, I., Zhang, X., and Xu, X. A Semantc-based Approach for Mnng Undscovered Publc Knowledge from Bomedcal Knowledge. n Proceedngs of IEEE Internatonal Conference on Granular Computng Bejng. 15. Pratt, W., and Yetsgen-Yldz, M. LtLnker: Capturng Connectons across the Bomedcal Lterature. n Proceedngs of Internatonal Conference on Knowledge Capture (K-Cap'03) Florda.

13 Evaluaton of Lterature-based Dscovery Systems Hrstovsk, D., Stare, J., Peterln, B., and Dzerosk, S. Supportng dscovery n medcne by assocaton rule mnng n Medlne and UMLS. n Proceedngs of Mednfo Yetsgen-Yldz, M., and Pratt, W., Usng Statstcal and Knowledge-Based Approaches for Lterature Based Dscovery. Journal of Bomedcal Informatcs (To appear), Wren, J.D., Extendng the mutual nformaton measure to rank nferred lterature relatonshp. BMC Bonformatcs, (1): p Weeber, M., Vos, R., Klen, H., de Jong - van den Berg, L.T.W., and Aronson, A.R., Generatng hypotheses by dscoverng mplct assocatons n the lterature: a case report of a search for new potental therapeutc uses for thaldomde. Journal of Amercan Medcal Informatcs Assocaton (3): p Wren, J.D., Bekeredjan, R., Stewart, J.A., Shohet, R.V., and Garner, H.R., Knowledge dscovery by automated dentfcaton and rankng of mplct relatonshps. Bonformatcs, (3): p Swanson, D.R., Atral fbrllaton n athletes: mplct lterature-based connectons suggest that overtranng and subsequent nflammaton may be a contrbutory mechansm. Medcal Hypotheses, (6): p Skeels, M.M., Hennng, K., Yetsgen-Yldz, M., and Pratt, W. Interacton Desgn for Lterature-Based Dscovery. n Proceedngs of the Internatonal Conference for Human- Computer Interacton (CHI'05) Portland, WA. 23. Smalheser, N.R., et. al., Collaboratve development of the Arrowsmth two node search nterface desgned for laboratory nvestgators. Journal of Bomedcal Dscovery and Collaboraton, (8).