Russell V. Lenth Department of Statistics and Actuarial Science The University of Iowa

Transcription

1 Russell V. Lenth Department of Actuarial The University of Iowa in Iowa: Impact of the National Academy of s Report October 29, 2010

2 Important things to know 1 Probabilistic/statistical can be subtle 2 Interpreting data can be subtle... and that s assuming you have good data 3 Obtaining good data is subtle Outline of the talk Dealing with probability (primarily) and statistics in the courtroom Recommendations of ASA in response to NAS report

3 Identification based on rare events Fallacy of reversed conditions Confusing... The [im]probability of A, knowing that B is true with The [im]probability of B, knowing that A is true Illustration This tissue phenotype occurs in only 1 in 10,000 males; so there is a 9,999/10,000 chance that it came from a female.

4 Identification based on rare events Fallacy of reversed conditions Confusing... The [im]probability of A, knowing that B is true with The [im]probability of B, knowing that A is true Illustration This tissue phenotype occurs in only 1 in 10,000 males; so there is a 9,999/10,000 chance that it came from a female. Wrong!!! Suppose for instance that the phenotype occurs in only 1/100,000 females. Then the tissue is 100, 000/10, 000 = 10 times as likely to be from a male (assuming equal numbers of males and females)

5 Prosecutor s and defense attorney s fallacies Suspect matches a semen stain with DNA that occurs in 1/7000 of men. Prosecutor s fallacy The chance is 1/7000 that someone other than the suspect left the stain Defense attorney s fallacy Since there are 1,400,000 men in this area, there are /7000 = 200 other men who could have left this stain.

6 Prosecutor s and defense attorney s fallacies Suspect matches a semen stain with DNA that occurs in 1/7000 of men. Prosecutor s fallacy The chance is 1/7000 that someone other than the suspect left the stain Defense attorney s fallacy Since there are 1,400,000 men in this area, there are /7000 = 200 other men who could have left this stain. The prosecutor is arguing there is only one possible other perpetrator.

7 Prosecutor s and defense attorney s fallacies Suspect matches a semen stain with DNA that occurs in 1/7000 of men. Prosecutor s fallacy The chance is 1/7000 that someone other than the suspect left the stain Defense attorney s fallacy Since there are 1,400,000 men in this area, there are /7000 = 200 other men who could have left this stain. The prosecutor is arguing there is only one possible other perpetrator. The defense attorney is arguing that all 1,400,000 men are possible perpetrators.

8 Kingston s law (JASA 1965) Takes the suspect-population size, n, into account (suspect and n 1 others) When match probability p is low and n is low, the probability that the suspect left the trace evidence is approximately 1 np/4 Example (n = 5): /4 = /7000 Probability that suspect uniquely matches the evidence is approximately 1 np/2 Example (n = 5): /2 = 1 2.5/7000 Both are incorrect include some cases where suspect does not match! (Lenth, JFSS, 1986) Additional example of how subtle these things can be.

9 Sorting things out A picture always helps (Note: this is not to scale) Red areas represent possibilities of matching We know somebody matches so we re in the red area total = 1 + 4/ P(it is the suspect) = 1/ =

10 General case Can use same idea even when match with suspect is uncertain

11 General case Can use same idea even when match with suspect is uncertain Note how it changes with smaller population of suspects, or higher chances of matching

12 General case (more) Can use same idea to resolve reversed-conditions fallacy

13 significance... is just like a jury finding 1 Lots of damning evidence convict 2 Lots of exonerating evidence acquit 3 Inadequate (or bungled) evidence acquit

14 significance... is just like a jury finding 1 Lots of damning evidence convict 2 Lots of exonerating evidence acquit 3 Inadequate (or bungled) evidence acquit Similarly, 1 Lots of data suggesting an effect statistically significant 2 Lots of data suggesting no effect non-significant 3 Not much data (or botched methodology) non-significant Two very distinct reasons for acquitting (nonsignificance) Absence of proof (3) is not the same as proof of absence (2)

15 significance... is just like a jury finding 1 Lots of damning evidence convict 2 Lots of exonerating evidence acquit 3 Inadequate (or bungled) evidence acquit Similarly, 1 Lots of data suggesting an effect statistically significant 2 Lots of data suggesting no effect non-significant 3 Not much data (or botched methodology) non-significant Two very distinct reasons for acquitting (nonsignificance) Absence of proof (3) is not the same as proof of absence (2) There is another way to prove an effect is small

16 Some key findings/recommendations in NAS report Lack of sufficient recognition for sources of variation and uncertainties in forensic-science analysis Establish an institute of forensic science Independent federal agency Strongly rooted in science Strong ties to Academic and research communities Strong ties to state/local forensic science entities & professional organizations Not part of a law-enforcement agency Sufficient resources and prominence to make a difference Led by skilled and experienced people

17 Statisticians can help in this mission! (ASA Position Statement, May 3, 2010) Establish measurement protocols Quantify uncertainty Design experiments for testing new protocols or methodologies Analyze data from such experiments

18 issues for FS institute to address Assessment of forensic practices using properly designed experiments and data analytic methods Use of established statistical principles and procedures, including estimated error rates Review of novel methods in refereed journals; reviewers should include statisticians Employment of statistical quality control and quality assurance procedures Proficiency tests that employ accepted statistical designs, double-blind when possible Availability of all expert and sufficient supporting data to permit independent review

19 Assessment of forensic practices Donald Kennedy: It s not that fingerprint analysis is unreliable. The problem, rather, is that its reliability is unverified... Scientific evidence whose reliability may (will?) be exaggerated when presented to a jury e.g., FBI study of 50,000 fingerprints: Unpublished study prepared specifically for litigation Compares a digitized fingerprint with itself rather than other prints of the same finger Unrealistic estimates of standard error Poor statistical modeling

20 Assessment of forensic practices Donald Kennedy: It s not that fingerprint analysis is unreliable. The problem, rather, is that its reliability is unverified... Scientific evidence whose reliability may (will?) be exaggerated when presented to a jury e.g., FBI study of 50,000 fingerprints: Unpublished study prepared specifically for litigation Compares a digitized fingerprint with itself rather than other prints of the same finger Unrealistic estimates of standard error Poor statistical modeling How you collect data is often much more important than how you analyze data

21 Use of established statistical procedures Validity of methods should be demonstrated... and published in peer-reviewed journals Example: Compositional analysis of bullet lead (CABL) Assumes that chemical composition provides a unique signature for a batch of bullets FBI validation claims based on matching bullets in 1837 cases with experimental bullets Selection claimed to be random but is demonstrably not Not a representative selection either test used was ad hoc

22 Use of established statistical procedures Validity of methods should be demonstrated... and published in peer-reviewed journals Example: Compositional analysis of bullet lead (CABL) Assumes that chemical composition provides a unique signature for a batch of bullets FBI validation claims based on matching bullets in 1837 cases with experimental bullets Selection claimed to be random but is demonstrably not Not a representative selection either test used was ad hoc R.A. Fisher: Sometimes the only thing you can do with a poorly designed experiment is to try to find out what it died of.

23 Rigorous review of new data-analysis methods In CABL study, use of ad hoc data-clustering method, clustered together bullets of very different compositions but claimed to have come from the same batch. High rate of false matches. DNA profiling: No consensus yet on analysis of complex mixtures of DNA; 15-year-old STR technology suffers from allelic dropout Int l Soc. Genetics report: let s hear from scientific community before recommending on mixture interpretation Nonetheless, crime-lab technicians often make strong and unqualified statistical statements based on ad hoc and unsupported methods.

24 Rigorous review of new data-analysis methods In CABL study, use of ad hoc data-clustering method, clustered together bullets of very different compositions but claimed to have come from the same batch. High rate of false matches. DNA profiling: No consensus yet on analysis of complex mixtures of DNA; 15-year-old STR technology suffers from allelic dropout Int l Soc. Genetics report: let s hear from scientific community before recommending on mixture interpretation Nonetheless, crime-lab technicians often make strong and unqualified statistical statements based on ad hoc and unsupported methods. My consulting experience: most scientists are unrealistically optimistic about the reliability and repeatability of their procedures

25 Modern statistical quality control Need for high-quality measurements, standardized procedures Clinical Laboratory Improvement Amendments (1988) establish quality standards for all laboratory testing and timeliness of patient test. laboratories are specifically excluded in CLIA Some regulation exists, e.g. Coll. Amer. Pathologists regulates forensic pathology We live in a variation-filled world. QC methods separate special causes of variation from common causes. We need to do lab studies to quantify these.

26 Modern statistical quality control Need for high-quality measurements, standardized procedures Clinical Laboratory Improvement Amendments (1988) establish quality standards for all laboratory testing and timeliness of patient test. laboratories are specifically excluded in CLIA Some regulation exists, e.g. Coll. Amer. Pathologists regulates forensic pathology We live in a variation-filled world. QC methods separate special causes of variation from common causes. We need to do lab studies to quantify these. ASA T-shirt: Uncertainty the only thing you can count on

27 Double-blind proficiency testing CTS fingerprint-proficiency test (1995): 68 of 156 respondents (44%) correctly identified five latent impressions and noted two eliminations Before then, there was very little proficiency testing Comments on 2006 CTS cartridge case test: too easy, not reflective of actual casework; not blind in any way Drór, Charlton, Péron (2006): Five fingerprint experts tested on fingerprints they had previously identified as matches, but were given a different story. Three found no match, one was not sure. Contextual bias

28 Double-blind proficiency testing CTS fingerprint-proficiency test (1995): 68 of 156 respondents (44%) correctly identified five latent impressions and noted two eliminations Before then, there was very little proficiency testing Comments on 2006 CTS cartridge case test: too easy, not reflective of actual casework; not blind in any way Drór, Charlton, Péron (2006): Five fingerprint experts tested on fingerprints they had previously identified as matches, but were given a different story. Three found no match, one was not sure. Contextual bias Sherlock Holmes: It is a capital mistake to theorize before one has data.

29 Public availability of expert FBI RFLP population database in early 1990s primary basis for published analyses justifying casework calculations. Database existed solely for use in statistical analyses; not an offender or forensic casework database Request by a statistician to use original data rather than data than that provided to attorneys; FBI DNA expert Budlowe informed him that FBI had made commitments to others, your analysis must not conflict with theirs Data may be used only in collaboration with Dr. Budlowe Use of data restricted to only this one paper All authors must agree before submission to a journal

30 Public availability of expert FBI RFLP population database in early 1990s primary basis for published analyses justifying casework calculations. Database existed solely for use in statistical analyses; not an offender or forensic casework database Request by a statistician to use original data rather than data than that provided to attorneys; FBI DNA expert Budlowe informed him that FBI had made commitments to others, your analysis must not conflict with theirs Data may be used only in collaboration with Dr. Budlowe Use of data restricted to only this one paper All authors must agree before submission to a journal There are three kinds of liars: liars, damned liars, and liars posing as statisticians

31 Cited references American Association (2010) American Association Supports major reforms of the nation s forensic science system, Press release, May 3, 2010, pdfs/ Endorsement.pdf. Drór, I.I., Charleton, D., Péron, A.E. (2006) Contextual information renders experts vulnerable to making erroneous identifications, International, 156, Food and Drug Administration (1988) Clinical Laboratory Improvement Amendments (CLIA), http: // ucm htm Gill, P., et al. (2006) DNA commission of the International Society of Genetics: recommendations on the interpretation of mixtures, International, 160, Giannelli, P.C. (1997) The DNA story: an alternative view [Book review of And the blood cried out by Harlan Levy], The Journal of Criminal Law and Criminology, 88, Grieve, D. (1996) Possession of truth, Journal of Identification, 46, 521. Kaye, D.H. (2003) Questioning a courtroom proof of the uniqueness of fingerprints, International Review, 71, Kennedy, D. (2003) science: oxymoron?,, 302, Kingston, C.R. (1965a) Applications of probability theory in criminalistics, Journal of the American Association, 60, (1965b) Applications of probability theory in criminalistics II, Journal of the American Association, 60, Lenth, R.V. (1986) On identification by probability, Journal of the Society, 26, National Research Council (2004) analysis: edu/catalog.php?record_id= weighing bullet lead evidence, Schwartz, A. (2008) Challenging firearms and toolmark identification part two, The Champion, XXXII, Spiegelman, C.H. and Kafadar, K. (2006) Data integrity and the scientific method: the case of bullet lead data as forensic evidence, Chance, 19, Related reading Aitken, C.G.G. (2005) the evaluation of evidence for forensic scientists, 2nd ed, New York: John Wiley & Sons. Finkelstein, M.O. (2001) Statistics for lawyers, 2nd ed., New York: Springer-Verlag. Finkelstein, M.O. (2009) Basic concepts in probability and statistics in the law, New York: Springer-Verlag. Gastwirth, J. (2000) science in the courtroom, New York: Springer-Verlag. Zeisel, H. and Kaye, D. (1997) Prove it with figures, New York: Springer-Verlag. Web access to this handout For a limited time after October 29, 2010, this handout may be obtained electronically at uiowa.edu/~rlenth/lenth-stat--sci-handout.pdf. 4