Re-examining the Effectiveness of Manual Review

Size: px
Start display at page:

Download "Re-examining the Effectiveness of Manual Review"

Transcription

1 Re-examining the Effectiveness f Manual Review William Webber Cmputer Science and Sftware Engineering The University f Melburne Victria 3010, Australia wew@csse.unimelb.edu.au ABSTRACT Recent studies have fund that autmated retrieval methds in e- discvery are nt nly cheaper than manual review, but are als as r mre reliable. We argue that these studies, while suggestive, are nt cnclusive. There is a high variability in the quality f unsupervised manual reviewers, as data frm the TREC Legal Track shws. The best manual reviewers are as reliable as autmated methds, and a prperly supervised manual review may be mre reliable than autmatin. We shw the effectiveness f the simple review management apprach f bserving the prprtins fund relevant between reviewers. Finally, we describe the experimental prtcl necessary fr a mre cnclusive cmparisn f manual and autmated review. 1. INTRODUCTION The vlume f electrnically-stred infrmatin (ESI) held by mdern crpratins is driving discvery t use varius frms f technlgy-assisted r autmated review. An imprtant questin is whether autmated methds are merely a cheaper but lwer-quality alternative t full manual review, r whether autmatin leads t dcument prductins f equal r even higher quality. The frmer alternative means autmatin is a cmprmise; the latter wuld make full manual review bslete. Tw recent studies have cmpared the quality f autmated retrieval and manual review, ne by a re-review f an earlier manual prductin [Ritblat et al., 2010], the ther thrugh an analysis f data frm the TREC 2009 Legal Track [Grssman and Crmack, 2011]. The frmer study finds autmated retrieval t be at least as cnsistent as manual review, while the latter cncludes that autmatin gives superir reliability. We revisit the cmparisn f autmated and manual review methds, and argue that the previus studies, thugh suggestive, are nt cnclusive. In particular, we re-examine the TREC Legal Track data, bserving that the reviewers used are f highly variable reliability. The best reviewers are f cmparable r better quality than the best autmated systems, even under the asymmetric experimental cnditins f the track. It is still pen t questin, therefre, whether an autmated system can surpass r even achieve the reliability f a prperly managed manual review team. Whether autmated tls have surpassed manual review in quality is a questin t imprtant t leave withut a firm answer. We therefre cnclude ur paper with what is required fr an experimental prgram t answer this questin mre cnclusively. Cpyright is held by the authr/wner(s). SIRE 11, July 28, 2011, Beijing. ACM SIGIR. 2. BACKGROUND It is well knwn that human assessrs frequently disagree n the relevance f a dcument t a tpic. Vrhees [2000] fund that experienced TREC assessrs, albeit wrking frm nly sentencelength tpic descriptins, had an average verlap (size f intersectin divided by size f unin) f between 40% and 50% n the dcuments they judged t be relevant. Vrhees cncludes that 65% recall at 65% precisin is the best retrieval effectiveness achievable, given the inherent uncertainty in human judgments f relevance. Bailey et al. [2008] survey ther studies giving similar levels f inter-assessr agreement. When ne cnceptin f relevance is authritative, assessrs d nt merely disagree; they make errrs. In legal discvery, the authritative cnceptin f relevance is that f the attrney verseeing the retrieval. The Interactive Task f the Legal Track f TREC includes such a tpic authrity, and prvides a prcess f appeal t this authrity fr uncvering assessr errrs (Sectin 2.3). The appeal results fr TREC 2009 fund that, n an assessment set in which 90% f dcuments were actually irrelevant, 33% f relevant assessments were in errr, as were 3% f irrelevant assessments [Hedin et al., 2009]. This is likely a lwer bund t the errr rate, since sme errrs may nt have been appealed (althugh cnversely sme appeals may have been errneusly upheld). Since assessrs disagree, and reviewers make mistakes, the prductin f a manual review prcess is nt an inerrant gld standard, which an autmated prcess might apprach but cannt surpass. The questin, rather, is whether a manual r an autmated review prcess gives mre reliable results. That is the tpic addressed by the studies described belw. 2.1 Terminlgy and measures Manual review dentes a prcess in which every candidate dcument fr prductin is reviewed fr relevance by at least ne human reviewer. Candidate dcuments might be every dcument in a crpratin s pssessin, but generally sme prir filtering has been perfrmed, by custdian fr instance, r by keywrd queries, thugh the latter blurs the line between manual and autmated review. Autmated review dentes a situatin in which the decisin t prduce r nt prduce sme prprtin f the candidate dcuments is made algrithmically, withut cmplete human review. The term technlgy-assisted review is ften used instead, but while this may be sfter t a judge s ears, it seems t us inexact and unhelpful; surely all review f ESI requires the assistance f at least sme degree f technlgy. We use three measures f a retrieval s effectiveness: precisin, recall, and the F1 scre. Precisin is the prprtin f retrieved dcuments that are relevant; recall is the prprtin f relevant dcuments that retrieved. There is a natural tensin between the tw

2 measures: shrinking the retrieved set generally helps precisin, but can nly decrease recall; expanding the retrieval can nly help recall, but generally hurts precisin. This tensin is captured in the F1 measure, which is the harmnic mean f precisin and recall: F 1 = 2 R P RP. (1) 2.2 Ritblat, Kershaw, and Ot [2010] The first study cmparing manual and autmated review that we cnsider is that f Ritblat et al. [2010]. Fr their study, the authrs revisit the utcme f an earlier, in-huse manual review. The riginal review surveyed a crpus f 2.3 millin dcuments in respnse t a regulatry request, and prduced 176,440 as respnsive t the request; the prcess tk fur mnths and cst almst $14 millin. Ritblat et al. had tw autmated systems and tw manual review teams review the dcuments again fr relevance t the riginal request. The autmated systems wrked n the entire crpus; the manual review teams lked at a sample f 5,000 dcuments. Ritblat et al. (Table 1) fund that the verlap between the relevance sets f the tw manual teams was nly 28%, even lwer than the 40% t 50% bserved in Vrhees [2000] fr TREC AdHc assessrs. The verlap between the new and the riginal prductins was als lw, 16% fr each f the manual teams, and 21% and 23% fr the autmatic systems. When cmpared against the riginal prductin, the human review teams achieved F1 scres f 7 and 8, while the autmated systems achieved 0.34 and The effectiveness scres calculated n the riginal prductin seemingly shw that the autmated systems are as reliable as the manual reviewers. Hwever, as Ritblat et al. nte, the riginal prductin is a questinable gld standard, since it likely is subject t the same variability in human assessment that the study itself demnstrates. Instead, the claim Ritblat et al. make fr autmated review is a mre cautius ne; namely, that tw manual reviews are n mre likely t prduce results cnsistent with each ther than an autmated review is with either f them. Given the remarkably lw level f agreement bserved by Ritblat et al., their cnclusin might seem a less than reassuring ne; an attrney might ask nt, which f these methds is superir, but, is either f these methds acceptable? Mre imprtantly, the study des nt address the attrney s fundamental questin: des autmated r des manual review result in a prductin that mre reliably meets the verseeing attrney s cnceptin f relevance? 2.3 The TREC legal track The Legal Track f TREC prvides an bjective envirnment in which t validate and cmpare different retrieval methds fr e-discvery [Barn et al., 2006]. Since t date n participant has perfrmed a fully manual review, there has nt been a direct cmparisn f autmated and manual review methds, thugh (as will be seen shrtly) Grssman and Crmack [2011] present a methd fr extracting such a cmparisn frm the TREC data. Of particular interest fr cmparing manual and autmated review is the track s Interactive Task. The task seeks (within experimental limits) t replicate the cnditins f a real-wrld retrieval. In particular, there is a tpic authrity (TA), wh plays the rle f the attrney verseeing the prductin, and whse cnceptin f relevance is authritative. Teams may cnsult with the TA while prducing their runs, and the TA instructs (thugh des nt directly supervise) the track s relevance assessrs. Teams may als appeal initial assessments t the TA fr adjudicatin, with the adjudicated assessments frming the fficial assessment set fr the task. The dataset used by Grssman and Crmack [2011], and by the current paper, cmes frm the TREC 2009 Interactive Task. Seven Tpic Bins Type Ass d Ass Rel Appl d Adj Rel t Student t Student t Prf nl t Prf nl t Student t Student t Prf nl Table 1: Summary f assessment fr the interactive task tpics f the TREC 2009 Legal Track. Reprted are number f cre bins; assessr type (law student r prfessinal reviewer); number f messages sampled and assessed; number f messages initially assessed relevant; number f assessments appealed; number f messages assessed relevant after adjudicatin. tpics were run that year; their assessment utcmes are summarized in Table 1. In the task s assessment scheme, messages are sampled frm strata defined by participating team s intersecting prductins, and als frm the bttm stratum f messages returned by n system; the latter stratum is sampled sparsely, giving each sampled message a significant weight in effectiveness estimates [Hedin et al., 2009]. Dcuments ( bdies and attachments) in the messages sampled fr assessment are assigned t sets called bins (clumn 2 f Table 1). Each bin is assessed by a single assessr; an assessr may (rarely) assess mre than ne bin. Mst bins are cre bins, t which messages are randmly assigned. A small number f supplementary bins, with differing assignment methds, are used t achieve special assessment gals. Assessrs were f tw types in 2009 (clumn 3 f Table 1): first, vlunteer law students; r secnd, prfessinal manual reviewers. Each bin was assigned enugh messages (summed in clumn 4) t make up 500 dcuments. The number f messages initially assessed relevant varies widely between tpics (clumn 5), as des the number f appeals (clumn 6). Since appealing was at the discretin f the participating teams, the latter variety culd be due either t the errrs f the assessrs, r t the thrughness f the teams. Hw cmplete the appeals were in detecting errrs in the initial assessments is cnsidered in Sectin Grssman and Crmack [2011] Grssman and Crmack [2011] re-analyze the interactive task as a cmparative evaluatin f manual and autmated review, by treating the assessrs as a manual review team, and evaluating their retrieval, alngside that f the autmated systems, against the adjudicated assessments. They select fr this cmparisn tw tpperfrming autmated systems: an industry system which we will name System I, and an academic ne, System A. The five tpics in which these systems participated were heavily appealed, in particular by these teams themselves, leading t gd cverage f assessr errrs r, perhaps, a re-alignment f the TA s cnceptin f relevance with the appealing team s. The utcme f the evaluatin perfrmed by Grssman and Crmack is shwn in Table 2. The autmated systems beat the manual review teams quite handsmely fr fur f the five tpics, and cme clse fr the fifth. On this shwing, autmated retrieval appears nt merely an adequate, but a superir, alternative t manual review. The analysis f Grssman and Crmack assumes that the adjudicated assessments are a reasnably accurate gld standard, in the authrs wrds. This in turn requires that the appeal prcess is

3 Tpic Team Rec Prec F1 t201 System A TREC (Law Students) t202 System A TREC (Law Students) t203 System A TREC (Prfessinals) t204 System I TREC (Prfessinals) t207 System A TREC (Prfessinals) Prprtin assessed relevant Pre adjudicatin Pst adjudicatin Table 2: Autmated and manual reviewer effectiveness. Evaluatin is against the adjudicated assessments, extraplated t the full crpus f messages. The best autmated team fr the selected tpics is cmpared t the manual review team cnstructed frm the initial assessments f the track assessrs. (Based upn Table 7 f Grssman and Crmack [2011]; values are recalculated.) bth reasnably cmplete and unbiased. Incmplete appeals wuld leave assessment errrs unfund, inflating the effectiveness f manual review. On the ther hand, appeals culd shift the tpic authrity s cnceptin f relevance twards a team s run, especially since (unlike the riginal assessments) they are accmpanied by written justificatins. Which f these tw effects is strnger is unclear. Mre imprtantly, the re-purpsed assessments are nt true manual review effrts. Hw representative they are f a prperly supervised manual review is the tpic f Sectin RECONSIDERING MANUAL REVIEW The previus sectin surveyed tw recent studies cmparing the reliability f manual and autmated review. Next, we re-examine the measurement f manual review effectiveness, lking in particular at the evidence prvided by the TREC 2009 Legal Interactive task. 3.1 Cmpleteness f appeal prcess First, what evidence d we have fr the cmpleteness f the appeals prcess, assumed by [Grssman and Crmack, 2011]? Since messages sampled fr assessment are randmly assigned t cre assessment bins, we shuld expect each bin t have the same prprtin f relevant messages, subject t randm variatin. 1 Unevenness in prprtins initially assessed relevant is evidence f assessr errrs, and cntinued unevenness after adjudicatin is evidence that the appeals prcess has failed t uncver all such errrs. The cnverse is nt necessarily true: prprtins culd be balanced same even if many assessr errrs exist, thugh this wuld be likely in practice nly if the assessrs as a grup had a cnsistent, thugh incrrect, cnceptin f relevance. We illustrate the analysis f prprtins assessed relevant, taking Tpic 201 as an example. Figure 1 shws the prprtin f messages in each cre bin fr this tpic that were assessed relevant, 1 The chesin wuld be even strnger if the assignment were perfrmed s that each bin received the same prprtin f dcuments frm each stratum, but this latter step was nt in fact enfrced. Nte that we rely n the simple randm sampling f messages in ur analysis, nt f dcuments; the latter are nt simple-randmly sampled, but are clustered by messages Figure 1: Prprtin f messages assessed relevant in each cre bin, prir t and after the appeal and adjudicatin prcess, fr Tpic 201. Tpic Bins Assessed Bin Prprtin χ 2 Adjudicated t ** 9.9 t ** 14.7 t ** 11.1 t ** 9.9 t ** 45.5 ** t ** ** t Table 3: Chi-squared statistic fr prprtins relevant acrss cre bins befre and after adjudicatin. Prprtins significantly uneven at p < 01 are marked with **. befre and after adjudicatin. The prprtins relevant in the initial assessment vary enrmusly; the mean prprtin is 12%, but ne assessr finds 31% f messages relevant, anther just 4%. This prvides clear evidence f many assessr errrs. On the ther hand, after appeal and adjudicatin, the mean prprtin falls t 7%, and the range shrinks t between 5% and 10%. We can test whether the relevance prprtin between bins is uneven t a statistically significant degree using a χ 2 test f prprtins. The null hypthesis is that reviewers are applying exactly the same cnceptin f relevance, and that variability in prprtins assessed relevant is due t sampling errr alne. The χ 2 statistic measures the rati between the bserved and the expected variability between prprtins (subject t the number f bservatins). The tw-tailed expected 95% range fχ 2 fr 12 bins is3.8 t21.9, fr 13 bins 4.4 t Values abve that range indicate significant unevenness; values belw wuld indicate suspicius evenness (suggesting, fr instance, that teams set ut t prduce the same prprtin relevant per bin, regardless f actual relevance). The bservedχ 2 statistics fr the TREC 2009 tpics, befre and after adjudicatin, are given in Table 3. Prir t adjudicatin, the assessments fr all tpics ther than Tpic 207 shw highly significant degrees f unevenness in prprtins relevant between bins. After adjudicatin, five tpics are nt significantly uneven, being the five tpics examined by Grssman and Crmack [2011]. The

4 appeal prcess appears t have been reasnably cmplete fr these tpics. Tpic 206 was nly lightly appealed, as Table 1 indicates, and highly significant unevenness remains; we can therefre regard that tpic s adjudicated assessments as a pr gld standard, and exclude the tpic frm further analysis. Tpic 205, in cntrast, was the mst heavily appealed tpic, and yet significant unevenness remains; either there were an extrardinary number f assessr errrs, r smething untward has ccurred with the assessment prcess. Still, the degree f unevenness is greatly reduced thrugh adjudicatin; we retain this tpic in ur subsequent analyses. The prprtins relevant f Tpic 207 s prfessinal review team shw the expected degree f evenness even befre adjudicatin. That this evenness is evidence f a gd review prcess is shwn by the high reliability the team achieves in the analysis f Grssman and Crmack [2011] (Table 2), and is further cnfirmed by the examinatin f the reliability f individual assessrs, belw. The ptential f the simple statistical analysis f evenness between prprtins as a tl fr review prcess cntrl is examined later. 3.2 Sample and ppulatin accuracy Sme f the manual reviewer reliability figures given in Table 2 are rather alarming; fr instance, that the review team fr Tpic 201 achieved a precisin f nly 5, returning nly ne actually relevant message in every twenty they judged relevant. This is nt the reliability bserved n the messages actually sampled, thugh; rather, it is the reliability extraplated t the full ppulatin. Unequal sampling emphasises bttm stratum assessments verturned n appeal. Fr instance, fr Tpic 201, frm ne in tw t ne in eight messages were sampled frm upper strata, but nly ne in three-hundred frm the bttm stratum. Each successful appeal carries up t 150 times the weight n the bttm stratum that it des n the upper nes. Of the 1,927 messages sampled frm the bttm stratum fr this tpic, 72 were fund relevant by the assessrs, but 71 f these assessments were appealed, and all 71 were verturned n appeal; this is why such lw precisin is reprted fr the reviewers in Table 2. The strng weight n these bttm-stratum appeals means that even a slight appeal-induced bias wuld greatly harm the apparent precisin f the reviewers, and bst the recall f the teams. Mrever, even if the figures are taken at face value, what is being simulated here is essentially an unsrted linear review f the full crpus, and the errrs f (presumably) inattentin that such a review wuld turn up. Such an exhaustive linear review might be prevented in practice by a pre-filtering by custdian r keywrd; and errrs f inattentin wuld be readily picked up by dual-assessment, particularly f assessed-relevant messages. Fr cmparisn with the extraplated reliability figures in Table 2, we recalculate in Table 4 bth team and reviewer accuracy n the pst-adjudicatin sample f messages alne, withut extraplating t the full ppulatin. The relative rdering f team and reviewer is the same as n the ppulatin (Table 2), with the best team better than the cmpsite f reviewers fr every tpic except Tpic 207. The perfrmance f the weaker review teams, hwever, is less extreme than under extraplatin. Fr instance, the team f student reviewers fr Tpic 201 scred a precisin f 5 and an F1 scre f 9 n the ppulatin, due t 71 f their 72 relevance assessments n the sparsely-sampled bttm stratum being verturned n appeal; judged n the sample nly, hwever, their precisin imprves t1, and their F1 scre t0.52. The extraplated reliability figures in Table 2 are nt simply wrng, nr are the sample figures in Table 4 simply crrect. The raw reliability figures given in the frmer case, hwever, need t be treated with sme cautin, due t the magnifying effect n errrs f Tpic Team Rec Prec F1 t201 System A TREC (Law Students) t202 System A TREC (Law Students) t203 System A TREC (Prfessinals) t204 System I TREC (Prfessinals) t207 System A TREC (Prfessinals) Table 4: Autmated and manual reviewer effectiveness, evaluated n the sampled assessments directly, withut extraplatin t the full crpus. Other details are as fr Table 2. sampling, the ptential fr appeal-induced adjudicatin bias, and the lack f simple quality-cntrl mechanisms. The setup f the assessment may nt be a fair representatin f an actual manual review. Nevertheless, fr the fllwing analysis, we will use the reviewer reliability figures as extraplated t the ppulatin. 3.3 Variability in reviewer reliability The reviewer reliability scres in Table 2 are averages acrss each team f assessrs. Figure 1 and Table 3 indicate that fr mst assessment teams, there is great variability in the prprtin f messages that each assessr finds relevant, which suggests that there may be similar variability in errr rates. In this sectin, we directly investigate variability in assessr reliability. Figure 2 shws the reliability f the review perfrmed in each bin, evaluated against the adjudicated assessments, and cmpares it t the perfrmance f the autmated systems identified by Grssman and Crmack [2011]. Fr every tpic but ne (Tpic 207), there is a great diversity between the reliability f different reviewers. Per-bin precisin ranges frm almst t appraching, and the range f recall values is ften wide. Only fr Tpic 203 des the best autmated system clearly utperfrm the best manual reviewer. As befre, the prfessinal manual review team fr Tpic 207 stands ut. Several reviewers utperfrm the best autmated system, and even the weaker individual reviewers have bth precisin and recall abve 0.5. The variability in reviewer reliability seen in Figure 2 suggests the imprtance f a prper review management prcess. The best reviewers generally match the best autmated systems, even amngst student reviewers. A prcess that brught all reviewers up t the standards f the best perfrmers, such perhaps as the prcess emplyed by the grup in Tpic 207, wuld seem t have the ptential t ffer equal r superir reliability t the best autmated methds. Just excluding the weaker reviewers wuld by itself significantly imprve review team reliability. The next sectin explres a simple mechanism fr achieving this. 3.4 Imprving review team quality There are many tls that can be emplyed t imprve the quality f a review prcess, sme t d with human factrs, thers invlving statistics. Dual assessment, fr instance, can help catch randm errrs f inattentin, while secnd review by an authritative reviewer such as the supervising attrney can crrect miscnceptins f relevance during the review prcess, and adjust fr

5 Precisin Precisin (a) Tpic 201 (b) Tpic 202 Precisin Precisin (c) Tpic 203 (d) Tpic 204 Precisin Precisin (e) Tpic 205 (f) Tpic 207 Figure 2: Assessr precisin and recall, extraplated t ppulatin, fr Tpics and Tpic 207. Each circle represents the reliability f a cre bin. The red crss in each figure except that fr Tpic 205 gives the perfrmance f the best autmated retrieval effrt, as listed in Table 2.

6 Tpic Reviewers Rec Prec F1 t201 All Excl t202 All Excl t203 All Excl t204 All Excl t205 All Excl t207 All Excl Table 5: Review team effectiveness, including and excluding reviewers with a disprprtinate number f relevant dcuments assessr errrs nce it is cmplete [Webber et al., 2010]. Explring the full range f prcess quality management techniques is beynd ur current scpe. It has already been bserved, thugh, that the prprtin assessed relevant is a simple indicatr f verall review cnsistency and quality. Des it als indicate individual reliability, pinting ut unreliable assessrs fr retraining r exclusin? We begin by relating the prprtin fund relevant in a bin with the reliability f that bin, as measured by F1 scre. The gal is t identify bins that are utliers in the prprtin f messages they find relevant. T d this, we take the median prprtin relevant acrss all bins (since the median is mre rbust t utliers than the mean), and determine which bins prduce relevance prprtins that are significantly different frm the median, at level p < 1 in a tw-tailed exact binmial test. Figure 3 cmpares the pre-adjudicatin bin prprtins relevant with F1 scres acrss the different TREC tpics, indicating which bins are significantly different frm the median prprtin relevant. Nte, first, the spread in prprtins relevant, particularly the remarkable dispersin fr Tpic 205, revealing a review prcess that was clearly nt in cntrl. The relatinship is nt unanimus, but the mre reliable bins tend t be thse clser t the median prprtin f messages relevant. In particular, significant divergence frm the median appears t be a partial, thugh nt infallible, indicatr f reviewer unreliability. A simple apprach t imprving review team quality is t exclude thse reviewers whse prprtin relevant are significantly different frm the median, and re-apprtin their wrk t the mre reliable reviewers. Table 5 reprts the change in review team reliability if this step is taken, cnsidering nly the dcuments falling int the nn-excluded bins (r, equivalently, assuming the wrk frm the excluded reviewers is re-apprtined evenly and perfrmed t the same standard as the rest f each reviewer s bin). In accrdance with ur previus bservatins, there is a general imprvement in reliability, thugh nt always a great ne. Fr every tpic in which a bin is excluded (every tpic, that is, except fr the cnsistently-reliable Tpic 207), the F1 scre f the pst-exclusin review team is higher than that f the riginal, smetimes by an appreciable margin. Precisin als generally rises, but in a cuple f cases recall falls, reflecting the fact that being verly generus in ne s assessments can help draw in relevant dcuments ne might therwise have missed. Fully excluding reviewers based slely n the prprtin f dcuments they find relevant is a crude technique. Nevertheless, the results f this sectin suggest that this prprtin is a useful, if nly partial, indicatr f reliability, ne which culd be cmbined with additinal evidence t alert review managers when their review prcess is diverging frm a cntrlled state. It may be that review teams with better prcesses, such at the team frm Tpic 207, already use such techniques. Therefre, they need t be cnsidered when a benchmark fr manual review quality is being established, against which autmatic techniques can be cmpared. 4. ASSESSING REVIEW METHODS Ritblat et al. and Grssman and Crmack have presented evidence fr the equal r greater reliability f autmated cmpared t full manual review. The frmer study, thugh, takes a manual review itself as the gld standard. The TREC experiments reanalyzed by Grssman and Crmack d use a human tpic authrity t measure prductin quality, smething which is mre representative f prfessinal practice. We have bserved in the Sectin 3, hwever, that the manual review pseud-teams frmed by re-purpsing the track assessrs are highly variable in quality, suggesting a lack f the quality cntrl and directin that might be expected in a true, prfessinal review effrt. What is needed are experiments cmparing autmated and manual appraches n an even fting (as in Ritblat et al.), evaluated against the bjective standard f a supervising tpic authrity (as in Grssman and Crmack). The authrity shuld drive bth prductins, n the ne tpic: prviding cding standards and supervisin t the manual team, and seed queries and relevance assessments t the autmated ne. Bth prcesses, particularly the manual review, shuld be cnducted accrding t industry standards. The same tpic authrity shuld then assess the quality f each prductin, bth fr cnfrmity t their wn cnceptin f relevance, and fr the amunt f effrt invlved in the prductin. N single experiment f this srt can be cmprehensive: there are a variety nt nly f autmated review methds, but als f manual prcess strategies; and, f curse, there are a multitude f ptential crpra and prductin requests. And even such a setup as this invlves a degree f unrealism and artificiality, since actual prductins are made in several, pssibly iterated, stages (extracting, culling, reviewing, redacting, cllating), and inevitably with a cmplex mix f manual and autmated prcesses. Nevertheless, such experiments, by directly cmparing the tw appraches n an equal fting, in a mre realistic envirnment, and against a representative bjective standard, will allw us t draw firmer cnclusins n the relative merits f the manual and autmated review. 5. CONCLUSIONS The riginal review frm which Ritblat et al. draw their data cst $14 millin, and tk fur mnths f 100-hur weeks t cmplete. The cst, effrt, and delay underline the need fr autmated review techniques, prvided they can be shwn t be reliable. Given the strng disagreement between manual reviews, even sme lss in review accuracy might be acceptable fr the efficiency gained. If, thugh, autmated methds can cnclusively be demnstrated t be nt just cheaper, but mre reliable, than manual review, then the chice requires n hesitatin. Mrever, such an achievement fr autmated text-prcessing technlgy wuld mark an epch nt just in the legal dmain, but in the wider wrld. Tw recent studies have examined this questin, and advanced evidence that autmated retrieval is at least as cnsistent as manual review [Ritblat et al., 2010], and in fact seems t be mre reli-

7 F1 scre F1 scre Prprtin assessed relevant Prprtin assessed relevant (a) Tpic 201 (b) Tpic 202 F1 scre F1 scre Prprtin assessed relevant Prprtin assessed relevant (c) Tpic 203 (d) Tpic 204 F1 scre F1 scre Prprtin assessed relevant Prprtin assessed relevant (e) Tpic 205 (f) Tpic 207 Figure 3: Assessr F1 scre and prprtin assessed relevant by bin fr Tpics and Tpic 207. Scres are extraplated t the ppulatin; prprtins assessed relevant are taken frm the sample. Bins with a prprtin relevant significantly different frm the median (p < 1) are shwn as red crsses; nn-significant bins are black circles.

8 able [Grssman and Crmack, 2011]. These results are suggestive, but (we argue) nt cnclusive as they stand. Fr the latter study in particular (leaving questins f ptential bias in the appeals prcess aside), it is questinable whether the assessment prcesses emplyed in the track truly are representative f a gd quality manual review prcess. We have prvided evidence f the greatly varying quality f reviewers within each review team, indicating a lack f prcess cntrl (unsurprising since fr fur f the seven tpics the reviewers were nt a genuine team). The best manual reviewers were fund t be as gd as the best autmated systems, even with the asymmetry in the evaluatin setup. The ne, prfessinal team that des manage greater internal cnsistency in their assessrs is als the ne team that, as grup, utperfrms the best autmated methd. We have als pinted ut a simple, statistically based methd fr imprving prcess cntrl, by bserving the prprtin f dcuments fund relevant by each assessr, and cunselling r excluding thse wh appear t be utliers. Abve all, it seems that previus studies (and this ne, t) have nt directly addressed the crucial questin, which is nt hw much different review methds agreed r disagree with each ther (as in the study by Ritblat et al. [2010]), nr even hw clse autmated r manual review methds turn ut t have cme t the tpic authrity s gld standard (as in the study by Grssman and Crmack [2011]). Rather, it is this: which methd can a supervising attrney, actively invlved in the prcess f prductin, mst reliably emply t achieve their verriding gal, t create a prductin cnsistent with their cnceptin f relevance. There is gd, thugh (we argue) s far incnclusive, evidence that an autmated methd f prductin can be as reliable a means t this end as a (much mre expensive) full manual review. Quantifying the tradeff between manual effrt and autmatin, and validating prtcls fr verifying the crrectness f either apprach in practice, are particularly relevant in the multi-stage, hybrid wrk-flws f cntemprary legal review and prductin. Given the imprtance f the questin, we believe that it merits the effrt f a mre cnclusive empirical answer. Ellen Vrhees. Variatins in relevance judgments and the measurement f retrieval effectiveness. Infrmatin Prcessing & Management, 36(5): , September William Webber, Duglas W. Oard, Falk Schler, and Bruce Hedin. Assessr errr in stratified evaluatin. In Prc. 19th ACM Internatinal Cnference n Infrmatin and Knwledge Management, pages , Trnt, Canada, Octber References Peter Bailey, Nick Craswell, Ian Sbrff, Paul Thmas, A. de Vries, and Emine Yilmaz. Relevance assessment: are judges exchangeable and des it matter? In Sung-Hyn Myaeng, Duglas W. Oard, Fabrizi Sebastiani, Tat-Seng Chua, and Mun-Kew Leng, editrs, Prc. 31st Annual Internatinal ACM SIGIR Cnference n Research and Develpment in Infrmatin Retrieval, pages , Singapre, Singapre, July Jasn R. Barn, David D. Lewis, and Duglas W. Oard. TREC-2006 legal track verview. In Ellen Vrhees and Lri P. Buckland, editrs, Prc. 15th Text REtrieval Cnference, pages 79 98, Gaithersburg, Maryland, USA, Nvember NIST Special Publicatin Maura R. Grssman and Grdn V. Crmack. Technlgy-assisted review in e-discvery can be mre effective and mre efficient than exhaustive manual review. Richmnd Jurnal f Law and Technlgy, 17(3):11:1 48, Bruce Hedin, Stephen Tmlinsn, Jasn R. Barn, and Duglas W. Oard. Overview f the TREC 2009 legal track. In Ellen Vrhees and Lri P. Buckland, editrs, Prc. 18th Text REtrieval Cnference, pages 1:4:1 40, Gaithersburg, Maryland, USA, Nvember NIST Special Publicatin Herbert L. Ritblat, Anne Kershaw, and Patrick Ot. Dcument categrizatin in legal electrnic discvery: cmputer classificatin vs. manual review. Jurnal f the American Sciety fr Infrmatin Science and Technlgy, 61(1):70 80, 2010.

PET FORM Planning and Evaluation Tracking ( Assessment Period)

PET FORM Planning and Evaluation Tracking ( Assessment Period) Divisin f: Behaviral Studies PET FORM Planning and Evaluatin Tracking (2010 2011 Assessment Perid) Persn Respnsible fr this Divisin: Jerry Mller Department f: Behaviral Sciences Persn Respnsible fr this

More information

EXPLORING THE PROCESS OF ASSESSMENT AND OTHER RELATED CONCEPTS

EXPLORING THE PROCESS OF ASSESSMENT AND OTHER RELATED CONCEPTS 1 SECTION 1 INTRODUCTION: EXPLORING THE PROCESS OF ASSESSMENT AND OTHER RELATED CONCEPTS The Nature Of Assessment The Definitin Of Assessment The Difference Between Testing, Measurement And Evaluatin Characteristics

More information

A pre-conference should include the following: an introduction, a discussion based on the review of lesson materials, and a summary of next steps.

A pre-conference should include the following: an introduction, a discussion based on the review of lesson materials, and a summary of next steps. NAU Mdel Observatin Prtcl The mdel prtcl was develped with supprt and expertise frm the Natinal Institute fr Excellence in Teaching (NIET) and is based in great part n NIET s extensive experience cnducting

More information

Reliability and Validity Plan 2017

Reliability and Validity Plan 2017 Reliability and Validity Plan 2017 Frm CAEP The principles fr measures used in the CAEP accreditatin prcess include: (a) validity and reliability, (b) relevance, (c) verifiability, (d) representativeness,

More information

DATA RELEASE: UPDATED PRELIMINARY ANALYSIS ON 2016 HEALTH & LIFESTYLE SURVEY ELECTRONIC CIGARETTE QUESTIONS

DATA RELEASE: UPDATED PRELIMINARY ANALYSIS ON 2016 HEALTH & LIFESTYLE SURVEY ELECTRONIC CIGARETTE QUESTIONS DATA RELEASE: UPDATED PRELIMINARY ANALYSIS ON 216 HEALTH & LIFESTYLE SURVEY ELECTRONIC CIGARETTE QUESTIONS This briefing has been specifically prepared fr the Ministry f Health t prvide infrmatin frm this

More information

P02-03 CALA Program Description Proficiency Testing Policy for Accreditation Revision 1.9 July 26, 2017

P02-03 CALA Program Description Proficiency Testing Policy for Accreditation Revision 1.9 July 26, 2017 P02-03 CALA Prgram Descriptin Prficiency Testing Plicy fr Accreditatin Revisin 1.9 July 26, 2017 P02-03 CALA Prgram Descriptin Prficiency Testing Plicy fr Accreditatin TABLE OF CONTENTS TABLE OF CONTENTS...

More information

Structured Assessment using Multiple Patient. Scenarios (StAMPS) Exam Information

Structured Assessment using Multiple Patient. Scenarios (StAMPS) Exam Information Structured Assessment using Multiple Patient Scenaris (StAMPS) Exam Infrmatin 1. Preparing fr the StAMPS assessment prcess StAMPS is an assessment mdality that is designed t test higher rder functins in

More information

The estimator, X, is unbiased and, if one assumes that the variance of X7 is constant from week to week, then the variance of X7 is given by

The estimator, X, is unbiased and, if one assumes that the variance of X7 is constant from week to week, then the variance of X7 is given by ESTIMATION PROCEDURES USED TO PRODUCE WEEKLY FLU STATISTICS FROM THE HEALTH INTERVIEW SURVEY James T. Massey, Gail S. Pe, Walt R. Simmns Natinal Center fr Health Statistics. INTRODUCTION In April 97, the

More information

FOUNDATIONS OF DECISION-MAKING...

FOUNDATIONS OF DECISION-MAKING... Table f Cntents FOUNDATIONS OF DECISION-MAKING... Errr! Bkmark nt Describe the decisin-making prcess pp.62-66... Errr! Bkmark nt Explain the three appraches managers can use t make decisins pp.67-70 Errr!

More information

Q 5: Is relaxation training better (more effective than/as safe as) than treatment as usual in adults with depressive episode/disorder?

Q 5: Is relaxation training better (more effective than/as safe as) than treatment as usual in adults with depressive episode/disorder? updated 2012 Relaxatin training Q 5: Is relaxatin training better (mre effective than/as safe as) than treatment as usual in adults with depressive episde/disrder? Backgrund The number f general health

More information

Frequently Asked Questions: IS RT-Q-PCR Testing

Frequently Asked Questions: IS RT-Q-PCR Testing Questins 1. What is chrnic myelid leukemia (CML)? 2. Hw des smene knw if they have CML? 3. Hw is smene diagnsed with CML? Frequently Asked Questins: IS RT-Q-PCR Testing Answers CML is a cancer f the bld

More information

The principles of evidence-based medicine

The principles of evidence-based medicine The principles f evidence-based medicine By the end f this mdule yu shuld be able t: Describe what evidence based medicine is Knw where t find quality evidenced based medicine n the internet Be able t

More information

CONSENT FOR KYBELLA INJECTABLE FAT REDUCTION

CONSENT FOR KYBELLA INJECTABLE FAT REDUCTION CONSENT FOR KYBELLA INJECTABLE FAT REDUCTION INSTRUCTIONS This is an infrmed cnsent dcument which has been prepared t help yur Dctr infrm yu cncerning fat reductin with an injectable medicatin, its risks,

More information

PROCEDURAL SAFEGUARDS NOTICE PARENTAL RIGHTS FOR PRIVATE SCHOOL SPECIAL EDUCATION STUDENTS

PROCEDURAL SAFEGUARDS NOTICE PARENTAL RIGHTS FOR PRIVATE SCHOOL SPECIAL EDUCATION STUDENTS PROCEDURAL SAFEGUARDS NOTICE PARENTAL RIGHTS FOR PRIVATE SCHOOL SPECIAL EDUCATION STUDENTS INTRODUCTION This ntice prvides an verview f the parental special educatin rights, smetimes called prcedural safeguards

More information

Cnsideratin fr Optimizatin: Optimizatin is a prgram transfrmatin technique, which tries t imprve the cde by making it cnsume fewer resurces (i.e. CPU, Memry) and deliver high speed. In ptimizatin, high-level

More information

Breast Cancer Awareness Month 2018 Key Messages (as of June 6, 2018)

Breast Cancer Awareness Month 2018 Key Messages (as of June 6, 2018) Breast Cancer Awareness Mnth 2018 Key Messages (as f June 6, 2018) In this dcument there are tw sectins f messages in supprt f Cancer Care Ontari s Breast Cancer Awareness Mnth 2018: 1. Campaign key messages

More information

Annual Assembly Abstract Review Process

Annual Assembly Abstract Review Process Annual Assembly Abstract Review Prcess AAHPM and HPNA cllabrate t review and select abstracts fr Annual Assembly. The cmmittees meet prir t the calls t review and update the Assembly bjectives (Planning

More information

This clinical study synopsis is provided in line with Boehringer Ingelheim s Policy on Transparency and Publication of Clinical Study Data.

This clinical study synopsis is provided in line with Boehringer Ingelheim s Policy on Transparency and Publication of Clinical Study Data. abcd Clinical Study Synpsis fr Public Disclsure This clinical study synpsis is prvided in line with Behringer Ingelheim s Plicy n Transparency and Publicatin f Clinical Study Data. The synpsis which is

More information

Assessment Field Activity Collaborative Assessment, Planning, and Support: Safety and Risk in Teams

Assessment Field Activity Collaborative Assessment, Planning, and Support: Safety and Risk in Teams Assessment Field Activity Cllabrative Assessment, Planning, and Supprt: Safety and Risk in Teams OBSERVATION Identify a case fr which a team meeting t discuss safety and/r safety planning is needed r scheduled.

More information

Appendix C. Master of Public Health. Practicum Guidelines

Appendix C. Master of Public Health. Practicum Guidelines Appendix C Master f Public Health Practicum Guidelines 0 Gergia State University, Schl f Public Health Master f Public Health Practicum Guidelines Fr mre infrmatin, cntact Jessica Hwell Pratt, MPH Practicum

More information

Human papillomavirus (HPV) refers to a group of more than 150 related viruses.

Human papillomavirus (HPV) refers to a group of more than 150 related viruses. HUMAN PAPILLOMAVIRUS This infrmatin may help answer sme f yur questins and help yu think f ther questins that yu may want t ask yur cancer care team; it is nt intended t replace advice r discussin between

More information

Statement of Work for Linked Data Consulting Services

Statement of Work for Linked Data Consulting Services A. Backgrund Infrmatin Statement f Wrk fr Linked Data Cnsulting Services The Natinal Library f Medicine (NLM), in Bethesda, Maryland, is a part f the Natinal Institutes f Health, US Department f Health

More information

Lee County Florida Income Guideline Chart

Lee County Florida Income Guideline Chart NEIGHBORHOOD STABILIZATION PROGRAM OF LEE COUNTY BUYER-RELATED QUESTIONS 1. Why is NSP beneficial t yur buyers? Three key advantages make the NSP Prgram especially attractive t eligible buyers: 1) Investrs

More information

Year 10 Food Technology. Assessment Task 1: Foods for Special Needs. Name: Teacher:

Year 10 Food Technology. Assessment Task 1: Foods for Special Needs. Name: Teacher: Year 10 Fd Technlgy Assessment Task 1: Fds fr Special Needs Name: Teacher: Due Date: Term 2, Week 1 Type f Task: Design Task Planning Fd Requirements Cllectin f Assessment: Submit in Class Assessment Plicy:

More information

Swindon Joint Strategic Needs Assessment Bulletin

Swindon Joint Strategic Needs Assessment Bulletin Swindn Jint Strategic Needs Assessment Bulletin Swindn Diabetes 2017 Key Pints: This JSNA gives health facts abut peple with diabetes r peple wh might get diabetes in Swindn. This helps us t plan fr medical

More information

2018 Medical Association Poster Symposium Guidelines

2018 Medical Association Poster Symposium Guidelines 2018 Medical Assciatin Pster Sympsium Guidelines Overview The 3 rd Annual student-run Medical Assciatin f the State f Alabama Research Sympsium will take place n Friday and Saturday, April 13-14 at the

More information

LEVEL OF CARE GUIDELINES: INTENSIVE BEHAVIORAL THERAPY/APPLIED BEHAVIOR ANALYSIS FOR AUTISM SPECTRUM DISORDER HAWAII MEDICAID QUEST

LEVEL OF CARE GUIDELINES: INTENSIVE BEHAVIORAL THERAPY/APPLIED BEHAVIOR ANALYSIS FOR AUTISM SPECTRUM DISORDER HAWAII MEDICAID QUEST OPTUM LEVEL OF CARE GUIDELINES: INTENSIVE BEHAVIORAL THERAPY / APPLIED BEHAVIOR ANALYSIS FOR AUTISM SPECTRUM DISORDER HAWAII MEDICAID QUEST LEVEL OF CARE GUIDELINES: INTENSIVE BEHAVIORAL THERAPY/APPLIED

More information

The Mental Capacity Act 2005; a short guide for the carers and relatives of those who may need support. Ian Burgess MCA Lead 13 February 2017

The Mental Capacity Act 2005; a short guide for the carers and relatives of those who may need support. Ian Burgess MCA Lead 13 February 2017 The Mental Capacity Act 2005; a shrt guide fr the carers and relatives f thse wh may need supprt Ian Burgess MCA Lead 13 February 2017 Agenda Overview f the MCA The 5 Principles and the legal definitin

More information

Accounting Assessment Report

Accounting Assessment Report Accunting 2014-2015 Assessment Reprt 1. Please give a brief verview f the assessment data yu cllected this year. This can be in any frm yu feel is apprpriate, such as a table, a shrt narrative f results,

More information

Record of Revisions to Patient Tracking Spreadsheet Template

Record of Revisions to Patient Tracking Spreadsheet Template Recrd f Revisins t Patient Tracking Spreadsheet Template Belw is a recrd f revisins made by the AIMS Center t the Patient Tracking Spreadsheet Template. The purpse f this dcument is t infrm spreadsheet

More information

Interpretation. Historical enquiry religious diversity

Interpretation. Historical enquiry religious diversity Name: Year 8 Histry Prject 3: D The Cmmnwealth Games Still Matter In The 21 st Century? Mdule: Date Set: Deadline: Descriptin f the task: The prject is split int three separate parts: The prject is split

More information

Module 6: Goal Setting

Module 6: Goal Setting Mdule 6: Gal Setting Objectives T understand the cncept f gal setting in Brief CBT T acquire skills t set feasible and apprpriate gals in Brief CBT What is gal setting, and why is it imprtant t set gals

More information

Completing the NPA online Patient Safety Incident Report form: 2016

Completing the NPA online Patient Safety Incident Report form: 2016 Cmpleting the NPA nline Patient Safety Incident Reprt frm: 2016 The infrmatin cntained within this dcument is in line with the current Data Prtectin Act (DPA) requirements. This infrmatin may be subject

More information

Training module 1: Summary

Training module 1: Summary Draft (Step 2) guideline ICH E9(R1) Estimands and Sensitivity Analysis in Clinical Trials Training mdule 1: Summary Addendum t ICH E9 Statistical Principles fr Clinical Trials ICH E9(R1) Expert Wrking

More information

AP Biology Lab 12: Introduction to the Scientific Method and Animal Behavior

AP Biology Lab 12: Introduction to the Scientific Method and Animal Behavior Name: AP Bilgy Lab 12: Intrductin t the Scientific Methd and Animal Behavir Overview In this lab yu will: -Observe an rganism and design an experiment t investigate their respnses t envirnmental variables.

More information

Chapter 3 Perceiving Ourselves and Others in Organizations

Chapter 3 Perceiving Ourselves and Others in Organizations Chapter 3 Perceiving Ourselves and Others in Organizatins We begin this chapter by lking at hw peple perceive themselves, that is, their selfcncept. The Self-Cncept refers t an individual s self-beliefs

More information

Seeking and Appraising Evidence

Seeking and Appraising Evidence EWMA Educatinal Develpment Prgramme Curriculum Develpment Prject Educatin Mdule: Seeking and Appraising Evidence Latest review: August 2012 Educatin Mdule: Seeking and Appraising Evidence ABOUT THE EWMA

More information

Cardiac Rehabilitation Services

Cardiac Rehabilitation Services Dcumentatin Guidance N. DG1011 Cardiac Rehabilitatin Services Revisin Letter A 1.0 Purpse The Centers fr Medicare and Medicaid Services (CMS) has detailed specific dcumentatin requirements fr Cardiac Rehabilitatin

More information

Campus Climate Survey

Campus Climate Survey Campus Climate Survey Executive Summary www.ecu.edu/ecyu 2016 A prject spnsred by the Office fr Equity and Diversity Executive Summary Prject Backgrund In FY 2013-2014, the Campus Climate Cmmissin prpsed

More information

Cancer Association of South Africa (CANSA)

Cancer Association of South Africa (CANSA) Cancer Assciatin f Suth Africa (CANSA) Fact Sheet and Psitin Statement n Cannabis in Suth Africa Intrductin Cannabis is a drug that cmes frm Indian hemp plants such as Cannabis sativa and Cannabis indica.

More information

RI International Peer Employment Training

RI International Peer Employment Training RI Internatinal Peer Emplyment Training In Peer Supprt, mind and heart matter. Since 2000, we have successfully recruited and trained Peer Supprt Specialists at RI Internatinal (frmerly Recvery Innvatins)

More information

CSHCN Services Program Benefits to Change for Outpatient Behavioral Health Services Information posted November 10, 2009

CSHCN Services Program Benefits to Change for Outpatient Behavioral Health Services Information posted November 10, 2009 CSHCN Services Prgram Benefits t Change fr Outpatient Behaviral Health Services Infrmatin psted Nvember 10, 2009 Effective fr dates f service n r after January 1, 2010, benefit criteria fr utpatient behaviral

More information

University of Rochester Course Evaluation Project. Ronald D. Rogge. Associate Professor. Ista Zahn. Doctoral Candidate

University of Rochester Course Evaluation Project. Ronald D. Rogge. Associate Professor. Ista Zahn. Doctoral Candidate University f Rchester Curse Evaluatin Prject Rnald D. Rgge Assciate Prfessr Ista Zahn Dctral Candidate Department f Clinical & Scial Sciences in Psychlgy 2 Prject Impetus The current nline administratin

More information

A Phase I Study of CEP-701 in Patients with Refractory Neuroblastoma NANT (01-03) A New Approaches to Neuroblastoma Therapy (NANT) treatment protocol.

A Phase I Study of CEP-701 in Patients with Refractory Neuroblastoma NANT (01-03) A New Approaches to Neuroblastoma Therapy (NANT) treatment protocol. SAMPLE INFORMED CONSENT A Phase I Study f CEP-701 in Patients with Refractry Neurblastma NANT (01-03) A New Appraches t Neurblastma Therapy (NANT) treatment prtcl. The wrd yu used thrughut this dcument

More information

Code of employment practice on infant feeding

Code of employment practice on infant feeding Cde f emplyment practice n infant feeding An Emplyer s guide t: Sectin 69Y f the Emplyment Relatins Act 2000 Frewrd As Minister f Labur, I am pleased t publish the Cde f Emplyment Practice n Infant Feeding.

More information

DIRECTED FORGETIING: SHORT-TERM MEMORY OR CONDITIONED RESPONSE? WENDY S. MILLER and HARVARD L. ARMUS The University of Toledo

DIRECTED FORGETIING: SHORT-TERM MEMORY OR CONDITIONED RESPONSE? WENDY S. MILLER and HARVARD L. ARMUS The University of Toledo The Psychlgical Recrd, 1999, 49, 211-220 DIRECTED FORGETIING: SHORT-TERM MEMORY OR CONDITIONED RESPONSE? WENDY S. MILLER and HARVARD L. ARMUS The University f Tled Previus researchers have interpreted

More information

Success Criteria: Extend your thinking:

Success Criteria: Extend your thinking: Discussin Directr Yur jb is t invlve thers in cnversatin abut the text by getting them t think and talk abut the BIG IDEAS in the chapter/ sectin they have just read. Cmpse 5 questins that yu want t discuss

More information

Assessment criteria for Primary Health Disciplines Eligibility for Recognition as Credentialled Diabetes Educator. December 2015 ADEA

Assessment criteria for Primary Health Disciplines Eligibility for Recognition as Credentialled Diabetes Educator. December 2015 ADEA Assessment criteria fr Primary Health Disciplines Eligibility fr Recgnitin as Credentialled Diabetes Educatr December 2015 ADEA ASSESSMENT CRITERIA FOR PRIMARY HEALTH DICIPLINES ELIGIBILITY FOR RECOGNITION

More information

TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK. John T. Cockrell System Development Corporation. and

TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK. John T. Cockrell System Development Corporation. and fto 7V7?*7 Technical Research Reprt 1171 LIBRARY TECHNICAL REPORT SECTION NAVAL POSTGRADUATE SCHOOt MONTEREY, CALIFORNIA 93940 AD TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK Jhn

More information

Corporate Governance Code for Funds: What Will it Mean?

Corporate Governance Code for Funds: What Will it Mean? Crprate Gvernance Cde fr Funds: What Will it Mean? The Irish Funds Industry Assciatin has circulated a draft Vluntary Crprate Gvernance Cde fr the Funds Industry in Ireland. 1. Backgrund On 13 June 2011,

More information

Methadone Maintenance Treatment for Opioid Dependence

Methadone Maintenance Treatment for Opioid Dependence POLICY STATEMENT Methadne Maintenance Treatment fr Opiid Dependence APPROVED BY COUNCIL: May 2010 PUBLICATION DATE: Dialgue, Issue 2, 2010 Disclaimer: As f May 19, 2018 physicians n lnger require an exemptin

More information

Benefits for Anesthesia Services for the CSHCN Services Program to Change Effective for dates of service on or after July 1, 2008, benefit criteria

Benefits for Anesthesia Services for the CSHCN Services Program to Change Effective for dates of service on or after July 1, 2008, benefit criteria Benefits fr Anesthesia Services fr the CSHCN Services Prgram t Change Effective fr dates f service n r after July 1, 2008, benefit criteria fr anesthesia will change fr the Children with Special Health

More information

How to become an AME Online

How to become an AME Online Hw t becme an AME Online 1. Check that yu meet the minimum technical requirements in rder t use the AME Online system: Operating System: Windws Vista (Service Pack 2) Windws 7 Windws 8, 8.1 Windws 10 Please

More information

Improving Surveillance and Monitoring of Self-harm in Irish Prisons

Improving Surveillance and Monitoring of Self-harm in Irish Prisons HSE Mental Health Divisin Stewart s Hspital, Palmerstwn, Dublin 20 Tel: 01 6201670 Email: inf@nsp.ie www.nsp.ie Imprving Surveillance and Mnitring f Self-harm in Irish Prisns Prject Scpe Dcument 8 th June

More information

Data Fusion for Predicting Breast Cancer Survival

Data Fusion for Predicting Breast Cancer Survival Data Fusin fr Predicting Breast Cancer Linbailu Jiang, Yufei Zhang, Siyi Peng Mentr: Irene Kaplw December 11, 2015 1 Intrductin 1.1 Backgrund Cancer is mre f a severe health issue than ever in ur current

More information

PREPARED FOR: U.S. Army Medical Research and Materiel Command Fort Detrick, Maryland

PREPARED FOR: U.S. Army Medical Research and Materiel Command Fort Detrick, Maryland AWARD NUMBER: W81XWH-14-1-0444 TITLE: Culd HER2 Hetergeneity Open New Therapeutic Optins in Patients with HER2- Primary Breast Cancer? PRINCIPAL INVESTIGATOR: Gary Ulaner, MD, PhD CONTRACTING ORGANIZATION:

More information

Related Policies None

Related Policies None Medical Plicy MP 3.01.501 Guidelines fr Cverage f Mental and Behaviral Health Services Last Review: 8/30/2017 Effective Date: 8/30/2017 Sectin: Mental Health End Date: 08/19/2018 Related Plicies Nne DISCLAIMER

More information

Service Change Process. Gateway 1 High-level Proposition. Innovation project name: Patient Self-Monitoring/Management of Warfarin

Service Change Process. Gateway 1 High-level Proposition. Innovation project name: Patient Self-Monitoring/Management of Warfarin Service Change Prcess Gateway 1 High-level Prpsitin Innvatin prject name: Patient Self-Mnitring/Management f Warfarin NHS Bury Please describe the service change being prpsed. Please describe what service(s)

More information

Health Consumers Queensland submission

Health Consumers Queensland submission Health Cnsumers Queensland submissin Inquiry int Public Health (Medicinal Cannabis) Bill 2016 Queensland Parliament Health, Cmmunities, Disability Services and Family Vilence Preventin Cmmittee Cntact:

More information

National Imaging Associates, Inc. (NIA) Frequently Asked Questions (FAQ s) For PA Health & Wellness Providers

National Imaging Associates, Inc. (NIA) Frequently Asked Questions (FAQ s) For PA Health & Wellness Providers Natinal Imaging Assciates, Inc. (NIA) Frequently Asked Questins (FAQ s) Fr PA Health & Wellness Prviders Questin GENERAL Why is PA Health & Wellness implementing a Medical Specialty Slutins Prgram? Answer

More information

Individual Assessments for Couples Treatment with HFCA

Individual Assessments for Couples Treatment with HFCA Individual Assessments fr Cuples Treatment with HFCA Jennifer S. Ripley, Ph.D. Many appraches t cuples therapy include an individual assessment whenever a cuple cmes fr treatment. Therapists shuld be aware

More information

MGPR Training Courses Guide

MGPR Training Courses Guide MGPR Training Curses Guide fiscal cde 92107050921 1. Descriptin The training prgram supprted by MGPR is prpsed by a grup f excellent mentrs/educatrs, accmplished in Pesticides Management and Analysis,

More information

Impacts of State Level Dental Hygienist Scope of Practice on Oral Health Outcomes in the U.S. Population

Impacts of State Level Dental Hygienist Scope of Practice on Oral Health Outcomes in the U.S. Population Impacts f State Level Dental Hygienist Scpe f Practice n Oral Health Outcmes in the U.S. Ppulatin Building Trust in Peple and Places C-spnsred by: Prfessinal Standards Authrity Centre fr Trust, Peace and

More information

Recommendations for Risk Management at Swine Exhibitions and for Show Pigs August 2012

Recommendations for Risk Management at Swine Exhibitions and for Show Pigs August 2012 Recmmendatins fr Risk Management at Swine Exhibitins and fr Shw Pigs August 2012 Backgrund: The Natinal Prk Bard facilitated in develping this dcument. These recmmendatins were develped by a wrking grup

More information

Building Code 101 OWMC November 20, Ministry of Municipal Affairs and Housing

Building Code 101 OWMC November 20, Ministry of Municipal Affairs and Housing Building Cde 101 OWMC Nvember 20, 2015 Ministry f Municipal Affairs and Husing Disclaimer These slides are prvided by the Ministry f Municipal Affairs and Husing fr cnvenience nly The slides shuld nt be

More information

Nutrition Care Process Model Tutorials. Nutrition Monitoring & Evaluation: Overview & Definition. By the end of this module, the participant will:

Nutrition Care Process Model Tutorials. Nutrition Monitoring & Evaluation: Overview & Definition. By the end of this module, the participant will: Nutritin Care Prcess Mdel Tutrials Nutritin Care Prcess and Terminlgy Cmmittee Academy f Nutritin and Dietetics Nutritin Care Prcess Terminlgy 2015 Editin Nutritin Mnitring & Evaluatin: Overview & Definitin

More information

REGISTERED REPORTS AUTHOR AND REVIEWER GUIDELINES

REGISTERED REPORTS AUTHOR AND REVIEWER GUIDELINES REGISTERED REPORTS AUTHOR AND REVIEWER GUIDELINES A Registered Reprt is a frm f empirical article ffered at Nature Human Behaviur in which the methds and prpsed analyses are pre-registered and reviewed

More information

Novel methods and approaches for sensing, evaluating, modulating and regulating mood and emotional states.

Novel methods and approaches for sensing, evaluating, modulating and regulating mood and emotional states. Nvel methds and appraches fr sensing, evaluating, mdulating and regulating md and emtinal states. 2018 Jy Academic Grant Call fr Prpsals Intrductin The Annual Jy grant initiative aims t prmte and cntribute

More information

Whither Driver Education and Training?

Whither Driver Education and Training? Whither Driver Educatin and Training? Smith, G..the nly methd f refrming human nature and f btaining a higher standard f mrality is by educating the mind f the individual and imprving the scial cnditins

More information

Field Epidemiology Training Program

Field Epidemiology Training Program Field Epidemilgy Training Prgram Cancer Curriculum: Principles f Cancer Registries Case Study: Hspital-Based Cancer Registries FACILITATOR GUIDE FETP Cancer Curriculum: Principles f Cancer Registries Case

More information

A catalogue of criteria to objectify the harm-benefit analysis according to Austrian legislation

A catalogue of criteria to objectify the harm-benefit analysis according to Austrian legislation LINZ 2013 18th Eurpean Cngress n Alternatives t Animal Testing EUSAAT 2013 15th Annual Cngress f EUSAAT 15 18 September 2013, University f Linz, Austria A catalgue f criteria t bjectify the harm-benefit

More information

National Imaging Associates, Inc. (NIA) Frequently Asked Questions (FAQs) For Managed Health Services (MHS)

National Imaging Associates, Inc. (NIA) Frequently Asked Questions (FAQs) For Managed Health Services (MHS) Questin GENERAL Why did MHS implement a Medical Specialty Slutins Prgram? Natinal Imaging Assciates, Inc. (NIA) Frequently Asked Questins (FAQs) Fr Managed Health Services (MHS) Answer Effective Nvember

More information

Annual Principal Investigator Worksheet About Local Context

Annual Principal Investigator Worksheet About Local Context Cmpleting the NCI CIRB Annual Principal Investigatr Wrksheet Abut Lcal Cntext and the Study-Specific Wrksheet Abut Lcal Cntext at the University f Iwa All investigatrs cnducting research with the Natinal

More information

STATISTICAL ANALYSIS PLAN. Addendum 1 (V 1 12SEP2015)

STATISTICAL ANALYSIS PLAN. Addendum 1 (V 1 12SEP2015) Pelvic Flr Disrder STATISTICAL ANALYSIS PLAN Addendum 1 (V 1 12SEP2015) PFDN Prtcl Number 22P01: Refractry Overactive Bladder: Sacral NEurmdulatin v. BTulinum Txin Assessment (ROSETTA) AUTHOR (S): Tracy

More information

The Interface Between Theory of Mind and Language Impairment

The Interface Between Theory of Mind and Language Impairment The Interface Between Thery f Mind and Language Impairment By Lauren Lwry Hanen Certified SLP and Clinical Staff Writer When mst f us hear the term thery f mind we usually think abut children with autism

More information

(Please text me on once you have submitted your request online and the cell number you used)

(Please text me on once you have submitted your request online and the cell number you used) Dear Thank yu fr yur email, nted. Belw steps n hw t register as a service prvider. Please nte that nce yu have requested t becme a service prvider, yu need t sms/what s up me n 0826392585, in rder t activate

More information

ACMPR - Access to Cannabis for Medical Purposes Regulations Part 1 Commercial Production

ACMPR - Access to Cannabis for Medical Purposes Regulations Part 1 Commercial Production - Access t Cannabis fr Medical Purpses Regulatins Part 1 Cmmercial Prductin September 2017 OVERVIEW Yu must cmply with regulatins t supply medical cannabis as f August 24, 2016 There are fur parts t the

More information

Catherine Worthingham Fellows of the APTA Instructions for Nominators

Catherine Worthingham Fellows of the APTA Instructions for Nominators Catherine Wrthingham Fellws f the APTA Instructins fr Nminatrs The fllwing infrmatin was prepared t aid nminatrs in making a nminatin fr the APTA membership categry f Catherine Wrthingham Fellw. Please

More information

This standard operating procedure applies to stop smoking services provided by North 51.

This standard operating procedure applies to stop smoking services provided by North 51. Authr Name/Title Melanie McIlvar, Bid Develpment Manager Authr Signature Date: 4 th September 2017 Apprver Name/Title Jasn Shelley, Grup Directr f QA/RA Apprver Signature Date: 4 th September 2017 Issue

More information

A Unified Approach to Conflict Mineral Compliance for the Tungsten Industry. The Westin, Sydney, 23 September 2013

A Unified Approach to Conflict Mineral Compliance for the Tungsten Industry. The Westin, Sydney, 23 September 2013 A Unified Apprach t Cnflict Mineral Cmpliance fr the Tungsten Industry The Westin, Sydney, 23 September 2013 Backgrund Cmpliance f public US dwnstream tungsten users with the reprting requirements f the

More information

Who is eligible for LifeCare? What services are available?

Who is eligible for LifeCare? What services are available? Wh is eligible fr LifeCare? What services are available? LifeCare is an emplyer prvided wrk/life benefit frm The University f Texas at Austin available t all benefits eligible emplyees and their husehld

More information

D E R B Y, D E R B Y S H I R E, N O T T I N G H A M & N O T T I N G H A M S H I R E L M I S U M M A R Y

D E R B Y, D E R B Y S H I R E, N O T T I N G H A M & N O T T I N G H A M S H I R E L M I S U M M A R Y D E R B Y, D E R B Y S H I R E, N O T T I N G H A M & N O T T I N G H A M S H I R E L M I S U M M A R Y 2 A B O U T This dcument prvides a summary f key findings fr the Derby, Derbyshire, Nttingham and

More information

23/11/2015. Introduction & Aims. Methods. Methods. Survey response. Patient Survey (baseline)

23/11/2015. Introduction & Aims. Methods. Methods. Survey response. Patient Survey (baseline) Intrductin & Aims Drug and Alchl Cnsultatin Liaisn (AOD CL) services aim t imprve identificatin and treatment f patients with AOD mrbidity. The csts and cnsequences f targeting AOD patients presenting

More information

Rate Lock Policy. Contents

Rate Lock Policy. Contents Rate Lck Plicy Cntents Rate Lcks... 2 Rate Lck Cnfirmatin... 2 Lck Term... 2 Pre-Lck... 2 Maximum Qualified Rate... 3 Extensins... 3 Cst t Extend... 3 Relcks... 4 Re-Negtiatin r Flat Dwn Plicy... 4 Prgram

More information

SUFFOLK COUNTY COUNCIL. Anti- Social Behaviour Act Penalty Notice. Code of conduct

SUFFOLK COUNTY COUNCIL. Anti- Social Behaviour Act Penalty Notice. Code of conduct SUFFOLK COUNTY COUNCIL Anti- Scial Behaviur Act 2003 Penalty Ntice Cde f cnduct SCC Penalty Ntice Cde f Cnduct & Administrative Guidance: revised August 2014 1 Cntents 1. Legal Basis 2. Purpse f the Penalty

More information

UNIT 6. DEVELOPING THREAT/HAZARD-SPECIFIC ANNEXES

UNIT 6. DEVELOPING THREAT/HAZARD-SPECIFIC ANNEXES UNIT 6. DEVELOPING THREAT/HAZARD-SPECIFIC ANNEXES This page intentinally left blank. UNIT INTRODUCTION Visual 6.1 This unit presents infrmatin n annexes that shuld be included in a schl emergency peratins

More information

Catherine Worthingham Fellows of APTA Instructions for Writing a Letter of Support

Catherine Worthingham Fellows of APTA Instructions for Writing a Letter of Support Catherine Wrthingham Fellws f APTA Instructins fr Writing a Letter f Supprt Fllwing is infrmatin designed t assist persns asked t write a letter f supprt fr a nminee fr the American Physical Therapy Assciatin

More information

The effects of a two-school. school-year. back education program. in elementary schoolchildren

The effects of a two-school. school-year. back education program. in elementary schoolchildren IEA Maastricht - 12 July 2006 The effects f a tw-schl schl-year multi-factrial back educatin prgram in elementary schlchildren Drs Elisabeth Geldhf Prf Dr D De Clercq Prf Dr I De Burdeaudhuij Prf Dr G

More information

ACSQHC National Consensus Statement: Essential Elements for High Quality End-oflife Care in Acute Hospitals.

ACSQHC National Consensus Statement: Essential Elements for High Quality End-oflife Care in Acute Hospitals. 27 March 2014 Prfessr Debra Picne Chief Executive Officer Australian Cmmissin n Safety and Quality in Health Care c/ Ms Jennifer Hill, Senir Prject Officer Level 5, 255 Elizabeth Street SYDNEY NSW 2000

More information

Podcast Transcript Title: Common Miscoding of LARC Services Impacting Revenue Speaker Name: Ann Finn Duration: 00:16:10

Podcast Transcript Title: Common Miscoding of LARC Services Impacting Revenue Speaker Name: Ann Finn Duration: 00:16:10 Pdcast Transcript Title: Cmmn Miscding f LARC Services Impacting Revenue Speaker Name: Ann Finn Duratin: 00:16:10 NCTCFP: Welcme t this pdcast spnsred by the Natinal Clinical Training Center fr Family

More information

Chapter 6: Impact Indicators

Chapter 6: Impact Indicators Overview Chapter 6: Impact Indicatrs The best measure f the lng-term impact f all HIV preventin activities is the HIV incidence rate, namely the number f new cases f HIV infectin per year divided by the

More information

Introduction. Lesson developed by: Heather Medema-Johnson, M.S, ATC, CSCS

Introduction. Lesson developed by: Heather Medema-Johnson, M.S, ATC, CSCS Updated DATE BSL PRO Lessn H35: Range f Mtin/Sit and Reach Lessn develped by: Heather Medema-Jhnsn, M.S, ATC, CSCS Intrductin A well-runded physical activity prgram is an integral part f a strng, verall

More information

Introduction to the Ethics (Overview of Major Theories- Part I)

Introduction to the Ethics (Overview of Major Theories- Part I) Intrductin t the Ethics (Overview f Majr Theries- Part I) Ahmed Kulanić Chapter Overview Intrductin Review f ethical theries Mst everyne shares cre values, desiring Life Happiness Intrductin Ability t

More information

Health Screening Record: Entry Level Due: August 1st MWF 150 Entry Year

Health Screening Record: Entry Level Due: August 1st MWF 150 Entry Year Health Screening Recrd: Entry Level MIDWIFERY EDUCATION PROGRAM HEALTH SCREENING REQUIREMENTS (Rev. June 2017) 1. Hepatitis B: Primary vaccinatin series (3 vaccines 0, 1 and 6 mnths apart), plus serlgic

More information

Public consultation on the NHMRC s draft revised Australian alcohol guidelines for low-risk drinking

Public consultation on the NHMRC s draft revised Australian alcohol guidelines for low-risk drinking Public cnsultatin n the NHMRC s draft revised Australian alchl guidelines fr lw-risk drinking Recmmendatins frm The Cancer Cuncil Australia The Cancer Cuncil Australia is Australia s peak nn-gvernment

More information

Dental Benefits. Under the TeamstersCare Plan, you and your eligible dependents have three basic options when you need dental care.

Dental Benefits. Under the TeamstersCare Plan, you and your eligible dependents have three basic options when you need dental care. Dental Benefits Under the TeamstersCare Plan, yu and yur eligible dependents have three basic ptins when yu need dental care. Optin #1: TeamstersCare Dentists. Yu can use ur in-huse Charlestwn, Chelmsfrd,

More information

GUIDANCE DOCUMENT FOR ENROLLING SUBJECTS WHO DO NOT SPEAK ENGLISH

GUIDANCE DOCUMENT FOR ENROLLING SUBJECTS WHO DO NOT SPEAK ENGLISH GUIDANCE DOCUMENT FOR ENROLLING SUBJECTS WHO DO NOT SPEAK ENGLISH Aurra Health Care s Research Subject Prtectin Prgram (RSPP) This guidance dcument will utline the prper prcedures fr btaining and dcumenting

More information

SUICIDE AND MENTAL ILLNESS IN SINGAPORE

SUICIDE AND MENTAL ILLNESS IN SINGAPORE Vl. 15, N. 3. SINGAPORE MEDICAL JOURNAL 191 September, 1974. SUICIDE AND MENTAL ILLNESS IN SINGAPORE By W. F. Tsi and B. H. Chia SYNOPSIS This is a study f 112 cases f suicides wh had a past histry f being

More information

Introduction Teaching Interpretation

Introduction Teaching Interpretation Intrductin Teaching Interpretatin AUTHOR: Kyle Vanderwall Grandville High Schl, Grandville, MI Intrductin The AP U.S. Histry Curriculum Framewrk defines interpretatin in the fllwing way: Interpretatin

More information

1100 Marie Mount Hall College Park, Maryland Tel: (301) Fax: (301)

1100 Marie Mount Hall College Park, Maryland Tel: (301) Fax: (301) UNIVERSITY SENATE 1100 Marie Munt Hall Cllege Park, Maryland 20742-7541 Tel: (301) 405-5805 Fax: (301) 405-5749 http://www.senate.umd.edu March 31, 2017 Jrdan Gdman Chair, University Senate 2208G Physical

More information