Geoffrey Stewart Morrison

Transcription

1 Basic Workshop on Forensic Statistics Introduction to logical reasoning for the evaluation of forensic evidence Geoffrey Stewart Morrison p(e H ) p p(e H ) d

2 These slides are available on the following page

3 Concerns Logically correct framework for evaluation of forensic evidence - ENFSI Guideline for Evaluative Reporting 205; NCFS Views on statistical statements 206 But what is the warrant for the opinion expressed? Where do the numbers come from? - Risinger at ICFIS 20 Demonstrate validity and reliability - Daubert 993; NRC Report 2009; FSR Codes of Practice 204; PCAST Report 206 Transparency - R v T 200 Reduce potential for cognitive bias - NIST/NIJ Human Factors in Latent Fingerprint Analysis 202 Communicate strength of forensic evidence to triers of fact

4 Paradigm Use of the likelihood-ratio framework for the evaluation of forensic evidence logically correct Use of relevant data (data representative of the relevant population), quantitative measurements, and statistical models transparent and replicable relatively robust to cognitive bias Empirical testing of validity and reliability under conditions reflecting those of the case under investigation, using test data drawn from the relevant population only way to know how well it works

5 Empirical Validation

6 Empirical Validation The President s Council ofadvisors on Science and Technology report published on 20 September 206 Forensic science in criminal courts: Ensuring scientific validity of feature-comparison methods

7 Empirical Validation Without appropriate estimates of accuracy, an examiner s statement that two samples are similar or even indistinguishable is scientifically meaningless: it has no probative value, and considerable potential for prejudicial impact. (p 6) the expert should not make claims or implications that go beyond the empirical evidence and the applications of valid statistical principles to that evidence. (p 6) Where there are not adequate empirical studies and/or statistical models to provide meaningful information about the accuracy of a forensic feature-comparison method, DOJ attorneys and examiners should not offer testimony based on the method. (p 9)

8 Empirical Validation neither experience, nor judgment, nor good professional practices (such as certification programs and accreditation programs, standardized protocols, proficiency testing, and codes of ethics) can substitute for actual evidence of foundational validity and reliability. The frequency with which a particular pattern or set of features will be observed in different samples, which is an essential element in drawing conclusions, is not a matter of judgment. It is an empirical matter for which only empirical evidence is relevant. Similarly, an expert s expression of confidence based on personal professional experience or expressions of consensus among practitioners about the accuracy of their field is no substitute for error rates estimated from relevant studies.

9 Empirical Validation For forensic feature-comparison methods, establishing foundational validity based on empirical evidence is thus a sine qua non. Nothing can substitute for it. (p 6)

10 Likelihood Ratio Framework

11 Likelihood ratio framework 906 retrial of Alfred Dreyfus Jean-Gaston Darboux, Paul Émile Appell, Jules Henri Poincaré

12 Likelihood ratio framework Adopted as standard for evaluation of DNA evidence in mid 990 s

13 Likelihood ratio framework Association of Forensic Science Providers (2009) - Standards for the formulation of evaluative forensic science expert opinion 3 signatories [from Aitken to Zadora] (20) - Expressing evaluative opinions: A position statement European Network of Forensic Science Institutes (205) - Guideline for evaluative reporting in forensic science US National Commission on Forensic Science (206) - Views on statistical statements in forensic testimony

14 Bayesian Reasoning

15 Imagine you are driving to the airport...

16 Imagine you are driving to the airport...

17 Imagine you are driving to the airport...

18 Imagine you are driving to the airport... initial probabilistic belief + evidence updated probabilistic belief higher? or lower?

19 Imagine you are driving to the airport... initial probabilistic belief + evidence updated probabilistic belief higher? or lower?

20 This is Bayesian reasoning It is about logic It is not about mathematical formulae or databases There is nothing complicated or unnatural about it It is the logically correct way to think about many problems Thomas Bayes? Pierre-Simon Laplace

21 Imagine you work at a shoe recycling depot... You pick up two shoes of the same size Does the fact that they are of the same size mean they were worn by the same person? Does the fact that they are of the same size mean that it is highly probable that they were worn by the same person?

22 Imagine you work at a shoe recycling depot... You pick up two shoes of the same size Does the fact that they are of the same size mean they were worn by the same person? Does the fact that they are of the same size mean that it is highly probable that they were worn by the same person? Both similarity and typicality matter

23 Imagine you are a forensic shoe comparison expert... suspect s shoe crime-scene footprint

24 Imagine you are a forensic shoe comparison expert... The footprint at the crime scene is size 0 The suspect s shoe is size 0 What is the probability of the footprint at the crime scene would be size 0 if it had been made by the suspect s shoe? (similarity) Half the shoes at the recycling depot are size 0 What is the probability of the footprint at the crime scene would be size 0 if it had been made by the someone else s shoe? (typicality)

25 Imagine you are a forensic shoe comparison expert... The footprint at the crime scene is size 4 The suspect s shoe is size 4 What is the probability of the footprint at the crime scene would be size 4 if it had been made by the suspect s shoe? (similarity) % of the shoes at the recycling depot are size 4 What is the probability of the footprint at the crime scene would be size 4 if it had been made by the someone else s shoe? (typicality)

26 Imagine you are a forensic shoe comparison expert... The footprint at the crime science and the suspect s shoe are both size 0 similarity / typicality = / 0.5 = 2 you are twice as likely to get a size 0 footprint at the crime scene if it were produced by the suspect s shoe than if it were produced by someone else s shoe - someone else selected at random from the relevant population

27 Imagine you are a forensic shoe comparison expert... The footprint at the crime science and the suspect s shoe are both size 4 similarity / typicality = / 0.0 = 00 you are 00 times more likely to get a size 4 footprint at the crime scene if it were produced by the suspect s shoe than if it were produced by someone else s shoe - someone else selected at random from the relevant population

28 Imagine you are a forensic shoe comparison expert... size 0 similarity size 4 similarity / typicality = / 0.5 = 2 / typicality = / 0.0 = 00 If you didn t have a database, could you have made subjective estimates at relative proportions of different shoe sizes in the population and applied the same logic to arrive at a conceptually similar answer?

29 Area?

30 similarity / typicality = likelihood ratio

31 Given that it is a cow, what is the probability that it has four legs? p( 4 legs cow ) =?

32 Given that it has four legs, what is the probability that it is a cow? p( cow 4 legs ) =?

33 Given two voice samples with acoustic properties x and x2, what is the probability that they were produced by the same speaker? p( same speaker acoustic properties x, x ) =?

34 p( same speaker acoustic properties x, x ) =? 2 p( same wearer shoe size x, footprint size x ) =? p( cow x legs ) =?

35 Bayes Theorem posterior odds p( same speaker acoustic properties x, x2) p( different speaker acoustic properties x, x ) 2 = p( acoustic properties x, x2 same speaker ) p( same speaker ) p( acoustic properties x, x2 different speaker ) p( different speaker ) likelihood ratio prior odds

36 Bayes Theorem initial probabilistic belief + evidence updated probabilistic belief higher? or lower?

37 However!!! The forensic scientist acting as an expert witness cannot give the posterior probability. They cannot give the probability that two speech samples were produced by the same speaker.

38 Why not? The forensic scientist does not know the prior probabilities. Considering all the evidence presented so as to determine the posterior probability of the prosecution hypothesis and whether it is true beyond a reasonable doubt (or on the balance of probabilities) is the task of the trier of fact ( judge, panel of judges, or jury ), not the task of the forensic scientist. The task of the forensic scientist is to present the strength of evidence with respect to the particular samples provided to them for analysis. They should not consider other evidence or information extraneous to their task.

39 Bayes Theorem posterior odds p( same speaker acoustic properties x, x2) p( different speaker acoustic properties x, x ) 2 = p( acoustic properties x, x2 same speaker ) p( same speaker ) p( acoustic properties x, x2 different speaker ) p( different speaker ) likelihood ratio prior odds

40 Bayes Theorem posterior odds p( same speaker acoustic properties x, x2) p( different speaker acoustic properties x, x ) responsibility of trier of fact 2 = p( acoustic properties x, x2 same speaker ) p( same speaker ) p( acoustic properties x, x2 different speaker ) p( different speaker ) likelihood ratio prior odds

41 Bayes Theorem posterior odds p( same speaker acoustic properties x, x2) p( different speaker acoustic properties x, x ) responsibility of trier of fact 2 = p( acoustic properties x, x2 same speaker ) p( same speaker ) p( acoustic properties x, x2 different speaker ) p( different speaker ) likelihood ratio prior odds responsibility of forensic scientist

42 Likelihood Ratio p( acoustic properties x, x same speaker ) 2 p( acoustic properties x, x different speaker ) 2 p( shoe size x, footprint size x same wearer) p( shoe size x, footprint size x different wearer) p( x legs cow ) p( x legs not a cow ) p( E H ) prosecution p( E H ) defence

43 Example Forensic scientist: You would be 4 times more likely to get the acoustic properties of the voice on the offender recording if it were produced by the suspect than if it were produced by some other speaker selected at random from the relevant population. Whatever the trier of fact s belief as to the relative probabilities of the same-speaker versus the different-speaker hypotheses before being presented with the likelihood ratio, after being presented with the likelihood ratio their relative belief in the probability that the voices on the two recordings belong to the same speaker versus the probability that they belong to different speakers should be 4 times greater than it was before.

44 The evidence is 4 time more likely given the same-speaker hypothesis than given the different-speaker hypothesis Before multiply this weight by 4 After 4 different same if before you believed that the same-speaker and different-speaker hypotheses were equally probable different same now you should believe that the same-speaker hypothesis is 4 times more probable than the different-speaker hypothesis

45 The evidence is 4 time more likely given the same-speaker hypothesis than given the different-speaker hypothesis Before different same if before you believed that the same-speaker hypothesis was 2 times more probable than the different-speaker hypotheses multiply this weight by 4 2 After different same now you should believe that the same-speaker hypothesis is 8 times more probable than the different-speaker hypothesis 8

46 The evidence is 4 time more likely given the same-speaker hypothesis than given the different-speaker hypothesis Before multiply this weight by 4 After different same if before you believed that the different-speaker hypothesis was 2 times more probable than the same-speaker hypotheses different same now you should believe that the same-speaker hypothesis is 2 times more probable than the different-speaker hypothesis

47 The evidence is 4 time more likely given the same-speaker hypothesis than given the different-speaker hypothesis 8 Before different same if before you believed that the different-speaker hypothesis was 8 times more probable than the same-speaker hypotheses multiply this weight by 4 8 After 4 different same now you should believe that the different-speaker hypothesis is 2 times more probable than the same-speaker hypothesis

48 Likelihood Ratio Calculation I discrete data

49 Ready to calculate a likelihood ratio? p( E H ) prosecution p( E H ) defence p( x legs cow ) p( x legs not a cow )

50 Discrete data: bar graph 0.8 cows not cows proportion legs

51 cows not cows p( 4 legs cow ) p( 4 legs not a cow ) proportion legs

52 cows not cows p( 4 legs cow ) p( 4 legs not a cow ) proportion = legs

53 Relevant Population

54 Imagine you are a forensic hair comparison expert... The hair at the crime scene is blond The suspect has blond hair What do you do?

55 Imagine you are a forensic hair comparison expert... The hair at the crime scene is blond The suspect has blond hair p( blond hair at crime scene suspect is source) p( blond hair at crime scene someone else is source)

56 Imagine you are a forensic hair comparison expert... The hair at the crime scene is blond The suspect has blond hair p( blond hair at crime scene suspect is source) p( blond hair at crime scene someone else is source) Someone else selected at random from the relevant population

57 Imagine you are a forensic hair comparison expert... The hair at the crime scene is blond The suspect has blond hair p( blond hair at crime scene suspect is source) p( blond hair at crime scene someone else is source) Someone else selected at random from the relevant population What is the relevant population?

58 Imagine you are a forensic hair comparison expert... The hair at the crime scene is blond The suspect has blond hair p( blond hair at crime scene suspect is source) p( blond hair at crime scene someone else is source) Someone else selected at random from the relevant population What is the relevant population? Stockholm Beijing You need to use a sample representative of the relevant population

59 A likelihood ratio is the answer to a specific question the prosecution and the defence hypotheses. defined by The defence hypothesis specifies the relevant population. The forensic scientist must make explicit the specific question they answered so that the trier of fact can: understand the question consider whether the question is an appropriate question understand the answer

60 Fallacies of Interpretation

61 Prosecutor s Fallacy Forensic Scientist: One would be one thousand times more likely to obtain the acoustic properties of the voice on the intercepted telephone call had it been produced by the accused than if it had been produced by some other speaker from the relevant population. Prosecutor: So, to simplify for the benefit of the jury if I may, what you are saying is that it is a thousand times more likely that the voice on the telephone intercept is the voice of the accused than the voice of any other speaker from the relevant population.

62 Prosecutor s Fallacy Forensic Scientist: One would be one thousand times more likely to obtain the acoustic properties of the voice on the intercepted telephone call had it been produced by the accused than if it had been produced by some other speaker from the relevant population. p( E H ) prosecution p( E H ) defence Prosecutor: So, to simplify for the benefit of the jury if I may, what you are saying is that it is a thousand times more likely that the voice on the telephone intercept is the voice of the accused than the voice of any other speaker from the relevant population. p( H E ) prosecution p( E ) H defence

63 Defence Attorney s Fallacy Forensic Scientist: One would be one thousand times more likely to obtain the measured properties of the partial latent finger mark had it been produced by the finger of the accused than if it had been produced by a finger of some other person. Defence Attorney: So, given that there are approximately a million people in the region and assuming initially that any one of them could have left the finger mark, we begin with prior odds of one over one million, and the evidence which has been presented has resulted in posterior odds of one over one thousand. One over one thousand is a small number. Since it is one thousand times more likely that the finger mark was left by someone other than my client than that it was left by my client, I submit that this evidence fails to prove that my client left the finger mark and as such it should not be taken into consideration by the jury. THE MATHEMATICS IS CORRECT: prior odds likelihood ratio = posterior odds ( / ) 000 = / 000

64 Large Number Fallacy Forensic Scientist: One would be one billion times more likely to obtain the measured properties of the DNA sample from the crime scene had it come from the accused than had it come from some other person in the country. Trier of Fact: One billion is a very large number. The DNA sample must have come from the accused. I can ignore other evidence which suggests that it did not come from him.

65 Likelihood Ratio Calculation II continuous data

66 Discrete data: bar graph cows not cows p( 4 legs cow ) p( 4 legs not a cow ) proportion = legs

67 C ontinuous data: histograms probability density functions (PDFs) (a) (b) rectangle width: rectangle width: (c) (d) rectangle width: rectangle width: 0.

68 C ontinuous data: histograms probability density functions (PDFs) (a) (b) rectangle width: rectangle width: (c) (d) μ = 00 σ = rectangle width: rectangle width: 0.

69 0.025 suspect model probability density population model x

70 0.025 LR = 0.02 / = suspect model probability density population model x offender value

71 Gaussian Mixture Models (GMMs) suspect model population model

72 x

73 Thank You