Pushing Out the Frontiers. Nick D. K. Petraco and Many Others John Jay College of Criminal Justice!

Transcription

1 Applications of in Forensic Science Pushing Out the Frontiers Nick D. K. Petraco and Many Others John Jay College of Criminal Justice

2 Outline Admissibility of Scientific Evidence is a problem Frye and the Daubert Standards How chemistry, engineering, math and computers can help forensic science Current Projects At John Jay: Petroleum Distillates (Fire Debris) Dust (Trace Evidence) Cartridge cases (Firearms)

3 Admissibility of scientific evidence Principal legal standards: Frye and Daubert Frye (1923) Testimony offered as scientific must...have gained general acceptance in the particular field in which it belongs. New York is still a Frye State

4 Frye and Daubert Daubert (1993)- Judges are the gatekeepers of scientific evidence. Must determine if the science is reliable Has empirical testing been done? Falsifiability Has the science been subject to peer review? Are there known error rates? Is there general acceptance? Federal Government and 26(-ish) States are Daubert States

5 Raising Standards with Data and Statistics DNA profiling the most successful application of statistics in forensic science. Responsible for current interest in raising standards of other branches in forensics. No protocols for the application of statistics to physical evidence. Our goal: application of objective, numerical computational pattern comparison to physical evidence

6 Statistical pattern comparison Modern algorithms are called machine learning Idea is to measure features of the physical evidence that characterize it What Statistics Can Be Used? Train algorithm to recognize major differences between groups of features while taking into account natural variation and measurement error.

7 Why? R is not a proprietary black box Open-source and totally transparent R maintained by a professional group of statisticians, and computational scientists From very simple to state-of-the-art procedures available Very good graphics for exhibits and papers R is extensible (it is a full scripting language) R has commercial versions too, Revolution R, S+

8 Fire Debris Analysis Casework Liquid gasoline samples recovered during investigation: Unknown history Subjected to various real world conditions. If an individual sample can be discriminated from the larger group, this can be of forensic interest. Gas-Chromatography Commonly Used to ID gas. Peak comparisons of chromatograms difficult and time consuming. Does eye-balling satisfy Daubert, or even Frye...????

9 Study Design This study was undertaken to examine the variability of gasoline components in Twenty liquid gasoline samples Samples from fire investigations in the New York City area All samples analyzed using Gas Chromatography-Mass Spectrometry Keto and Wineman target compounds Fifteen peaks were chosen in this study that represented the common components present in gasoline. Normalized GC-MS peak areas were utilized to test the discrimination potential of multiple multivariate methods for discrimination.

10 Chosen Peaks M. Gil

11 Use prcomp, lda (MASS) and rgl: 10D PCA-3D CVA HOO-CV correct classification rate: 100%.

12 Dust N. Petraco

13 Dust A 19 th Century German magistrate influenced by the writings of Sir Arthur Conan Doyle suggested that Dust and other traces be allowed in legal proceedings. Hans Gross N. Petraco

14 What can it tell you? It enables one to identify the people places and things involved in an event. It helps one to associate the people, places and things involved in an event. It can often tell a story. It can help one reconstruct the event.

15 Develop a simple method that enables you to identify the trace materials commonly found in dust samples Develop a simple generic data sheet (Tool) that allows you to quickly collect data on the trace materials commonly found in dust samples Convert the data sheet in to an Excel Spreadsheet and load into Write some analysis scripts Analyze the data N. Petraco

16 Conformal Prediction TheoryVovk et al. New, but has roots in 1960 s with Kolmogorov s ideas on randomness and algorithmic complexity. Can be used with any statistical pattern classification algorithm. Independent of data s underlying probability distribution. This is a very important property for forensic pattern recognition Well, sample should be I.I.D. For identification of patterns, method produces Confidence region at Level of confidence, 1- α Confidence: Measure of how likely I.D. procedure is to be correct Results are valid: P(identification error) α

17 3D PCA-Clustering can show potential for discrimination Use e1071, caret, pls and custom scripts: PCA-SVM 27D, refined bootstrap error rate estimate= 0.7%, 95% CI [0.0%,3.3 %] CPT 99% level of confidence I.D. Empirical Error rate = 0% Unique and correct ID intervals = 93.1% PLS-DA 35D refined Bootstrap error rate estimate= 0.8% 95% CI [0.0%,3.3%]

18 Tool Marks G. Petillo Known Match Comparisons 5/8 Consecutively manufactured chisels

19 Approach For Striated Tool Marks Obtain striation pattern profiles form 3D confocal microscopy

20 Primershear Glock 19 firing pin impression P. Diaczuk

21 3D confocal image of entire shear pattern

22 Shear marks on primer of two different Glock 19s

23 Mean total profile: Mean waviness profile: Mean roughness profile:

24 Primer Shear Primer shears (82-91 profiles) PCA-SVM, CPT at the 95% level of confidence Empirical error rate was 4.7% 90.7% of I.D. intervals were unique and correct 7% of I.D. intervals had more than 1 I.D. No uninformative intervals were returned PCA-SVM, HOO-CV Error rate estimate is 0.0%-4.4%, depending on the number of replicates PLS-DA, Bootstrap (>10 replicates only) 95% confidence interval for error rate: [0%, 0%] 95% confidence interval for average false positive rate: [0%, 0%] 95% confidence interval for average false negative rate: [0%, 0%] PLS-DA, HOO-CV Error rate estimate is 0.0%-4.3%, depending on the number of replicates Results so far are on par with expectations

25 3D PCA 36 Glocks, 1080 simulated and real primer shear profiles: 18D PCA-SVM, refined bootstrap gun I.D. error rate 0.3%, 95% CI [0%, 0.8%]

26 Empirical Bayes Bayes Rule: can we realistically estimate posterior error probabilities empirically/falsifiably?? Probability of no actual association given a test/algorithm indicates a positive ID ( ) = Pr ( t + S - ) Pr( t + ) Pr S - t + Pr( S - ) Perhaps. Genomics has spawned similar questions: What is the probability of no disease (S - ) given the differences in expression scores of thousands of genes.

27 Empirical Bayes Erfon s machinery for empirical Bayes two-groups model Efron 2007 Surprisingly simple S -, truly no association, Null hypothesis S +, truly an association, Non-null hypothesis z, a Gaussian random variate derived from a machine learning task to ID an unknown pattern with a group Scheme yields estimate of Pr(S - z) along with it s standard error Called the local false discovery rate (fdr) or posterior error probability Given a similarity score, fdr is an estimate of the probability that the computer is wrong in calling a match Catch: you need A LOT of z and they should be fairly independent and Pr(S - ) > 0.9

28 Empirical Bayes Machine learning algorithms dump out tons of scores measuring how much they think each unknown piece of evidence, matches with a known Platt SVM probability scores from Glock 19 ID study: Primer Shear # (unknowns) Gun ID (knowns)

29 Empirical Bayes Use these SVM Platt scores to form p-values Transform p-values with probit function Produces A LOT of z values The z are fairly independent Null (Non Match) Histogram (via HOO) z values Density Normal Q Q Plot Theoretical Quantiles Sample Quantiles Checking assumptions on z-scores of Glock 19 primer shears

30 Empirical Bayes Use locfdr: Gives a calibrated posterior association probability model Estimated Posterior Error Probilities (Local FDRs) Estimated Posterior Error Probilities (Local FDRs) Pr(S z) est Pr(S z) est This is the est. prob of no association This is an uncertainty in the estimate z value z value Computer outputs match for: unknown-known from Bob the burglar, falls here

31 Empirical Bayes For the Glock 19 primer shear study: Posterior error probs. (black circles) estimated by HOO-CV Pr(S z) est The SVM alg got these Primer shear IDs wrong z value

32 Just Getting Started: Things to Come Footwear Soil Wrenches Chisels Q.D. Tire Tracks Hair Blood Spatter Gun Shot Residue N. Petraco N. Petraco

33 Research Team: Mr. Peter Diaczuk Dr. Peter De Forest Ms. Carol Gambino Mr. Mark Gil Dr. James Hamby Dr. Thomas Kubic Off. Patrick McLaughlin Dr. Linton Mohammed Mr. Jerry Petillo Mr. Nicholas Petraco Dr. Peter A. Pizzola Dr. Graham Rankin Dr. Jacqueline Speir Dr. Peter Shenkin Mr. Peter Tytell Acknowledgements Helen Chan Manny Chaparro Julie Cohen Aurora Dimitrova Eric Gosslin Frani Kammerman Brooke Kammrath Loretta Kuo Dale Purcel Stephanie Pollut Rebecca Smith Elizabeth Willie Chris Singh Melodie Yu