The Practice of Firearm/Toolmarks and a Reasonable Path Forward

Transcription

1 The Practice of Firearm/Toolmarks and a Reasonable Path Forward Cliff Spiegelman, Texas A&M University Coauthor William A. Tobin Copyright Cliff Spiegelman 1

2 Three Parts Of The Talk What are Firearm Toolmarks What statements etc. does the scientific method allow? Explanation of types of error with examples Three families of experiments to start laying the statistical foundation for firearm/toolmarks Discussion Copyright Cliff Spiegelman 2

3 Firearm/toolmarks Firearm toolmarks examinations and comparisons are often used in investigations of homicides involving a firearm, spent bullets and/or cartridge cases that are recovered from crime scenes. Most frequently, one or more firearms are recovered during investigation of a shooting incident and typically submitted for forensic comparisons with bullets and/or cartridge cases recovered from the scene. The forensic practice used to associate or eliminate a particular firearm as the murder weapon is based on comparisons of characteristics imparted to bullets and cartridge cases during cycling of a cartridge through the gun, and is known as firearm/toolmarks examination. Copyright Cliff Spiegelman 3

4 Typical testimony If it is concluded that the submitted weapon matches the crime scene bullets, the firearm/toolmarks examiner typically testifies at trial that the crime scene bullets were fired from the gun to the exclusion of all other possible weapons, although sometimes to a practical certainty. Copyright Cliff Spiegelman 4

5 What Claims does the Scientific Method Support? It supports claims that have been reasonably verified by hypotheses tests and corroborated by others. This means repeatable and reproducible and robust reproducible (in the case of broad claims) results Repeatability: The same experimental set up gets the same results Reproducibility: Similar setups that use different examiners at different locations get the same experiments results Robust Reproducibility: Widely varied experimental setups get the supporting results Copyright Cliff Spiegelman 5

6 What the scientific method does not allow Without repeatability claims should be restricted to the (crime) lab and the original experiment. Nothing can be said about any other experiment. Without reproducibility claims should be restricted to the (crime) labs participating in the experiment and experiments using the same factors and in the same way. Without robust reproducibility claims cannot be extended beyond the specific factors used in the experiment. Copyright Cliff Spiegelman 6

7 Error According To The Apple Dictionary Technical: a measure of the estimated difference between the observed or calculated value of a quantity and its true value. Law: a mistake of fact or of law in a court's opinion, judgment or order. Phrases: see the error of one's ways realize or acknowledge one's wrongdoing. Copyright Cliff Spiegelman 7

8 Two Categories Of Error Used In This Presentation Systematic or sometimes referred to as bias Better know examples: Publication bias: publishing only positive results Different instrumentation as in different light sources (much more about this later) Random error Better know examples: Variation among examiners trained within the same school ('CMS' or classical) Variation in marks from the same firearm Copyright Cliff Spiegelman 8

9 More Bias Examples Underreported, summary dismissal, or unreported false IDs Missouri and other cases reported in Gun evidence often wide of scientific mark in the Austin American Statesman Anthony Ray Hinton exonerated after 30 years on death row Former Detroit firearm unit (~10% error rate) Blame the lab or examiner rather than the process Question? Any such cases in the FBI firearm/toolmark unit? Present publications in trade journals as publications in science journals Copyright Cliff Spiegelman 9

10 AAS Article Highlighted By AFTE Website Copyright Cliff Spiegelman 10

11 More Bias Examples Myopic view of results Not involve the broader scientific community in research investigations but keep them within 'house Myopic view of results spreads by publication bias etc.. Restrictions to visible light (more later) Not combining human with computer determinations Using tortured language to deny a practical error rate match to a practical certainty or practical impossibility of being wrong Copyright Cliff Spiegelman 11

12 More on Bias An example of publication bias on steroids Have a trade organization claim it s publishes a science journal and publish non statistically designed, not double blinded studies and other pseudo-experiments as broad based proof of efficacy. Copyright Cliff Spiegelman 12

13 More Random Error Within bullet brand variation in charge and hence variation in striations Within bullet brand variation in bullet hardness (antimony levels) and hence variation in striations Random variations in light output Locations of bullet damage from striking bone or buildings Within bullet brand variation in cartridge hardness and hence variation in ejector marks Copyright Cliff Spiegelman 13

14 More Random Error Within bullet brand variation in primer cup hardness and hence variation in firing pin impressions Cleanliness of weapon components Condition of lubrication system, nested within lubrication nature and regime. etc..

15 Copyright Cliff Spiegelman 15

16 Some Factors To Consider During Firearm/Toolmark Experiments Factors not considered or discussed, and sample size issues: Ammunition type Ammunition charge Cartridge case hardness Primer cup hardness Breechface hardness Firing pin hardness Different batches of Glocks Non-Glock firearms feeds and speeds of production (or, alternatively, economic conditions of the manufacturing environment) Copyright Cliff Spiegelman 16

17 Factors To Consider (Continued) Ids from firing pins Ids from ejector marks Ids from breechface marks Ids from combination of ejector, firing pin, and breechface marks Weapon cleanliness Participants experience as toolmarks examiners Participants asked to handle test as casework, but no measures of effectiveness for this instruction reported Break-in period for pistols Lubrication regime Condition of lubrication system, nested within lubrication regime Copyright Cliff Spiegelman 17

18 Temper of workpiece Factors (Continued) Microstructure of workpiece Finishing processes (assertion only by author that breechface is unaffected during all remaining operations but no description as to what constituted remaining operations for interpretation of possible metallurgical effects) Effect of checking random sample of test cartridges for clear marks by experimenter before proceeding with the experiment was not investigated. This cannot be done in practice, as criminals do not check to insure their cartridges are clearly identifiable before leaving them at crime scenes. Copyright Cliff Spiegelman 18

19 Still More Factors Batch and sample not chosen randomly; Two 9 mm Lugers were not chosen randomly Acceptable undersized ammunition Brand of ammunition Response measures Fabrication tooling materials, if any Fabrication tooling hardness Type of workpiece Alloy used for workpiece

20 Possible Experimental Approaches to begin Laying a Statistical Foundation for Firearm Toolmarks Three classes of experiments will be presented 1. To build trust and confidence and motivate SOP writing. To build a team that can function together. 2. To assess examiners abilities to differentiate class, subclass, and individual characteristics 3. To assess error rates and assess their dependence upon many factors. To assess the importance of many factors and variables Copyright Cliff Spiegelman 20

21 Experiments Continued The experimental approaches should proceed sequentially and under no circumstances should the third set of experiments be performed before the first two. The reason is that without an SOP and without understanding examiners ability to distinguish various types of marks, error assessment will not serve a scientific purposes well. They can serve got you goals. For example Copyright Cliff Spiegelman 21

22 1. Building Confidence And Trust And Starting A Detailed SOP Pick a moderately priced pistol and film test fires from it Send the same test fired bullets (or cartridge casings) and video to participants Ask participants if the bullet was fired from the test weapon and to give the complete list of marks used or that could be used to match the bullet to the test gun. A template (2 dimensional blueprint, if need be colors can be used to demonstrate a 3rd dimension) of the bullet should be provided for examiners to show what marks that they used. Lands and groves should be numbered so that results from different examiners can be compared Copyright Cliff Spiegelman 22

23 Experiment Type 1 Continued After the results are returned summaries should be sent to all participants for discussion Discrepancies should be discussed Follow on experiments could remove some lands and groves from the original test bullets and the experiment repeated Eventually an SOP starts developing Copyright Cliff Spiegelman 23

24 2. Assess Examiners Abilities to Differentiate Class, Subclass, and Individual Characteristics (We use bullets but a similar approach can be used for cartridge casings) Select a few production runs and choose 2 consecutively manufactures runs from each. The runs should be chosen to cover different types of subclass characteristics and unique identifying marks (dirty lubrication vs. clean lubrication) Before each firing of the pistols make a map of each pistol barrel (using cameras and microscopy) Film test fires from each pistol and label them to the proper pistol and make video available to participating examiners Copyright Cliff Spiegelman 24

25 Experiment Type 2 Continued Provide a diagram to the FTE of all marks on the bullets. Ask them to label which ones are class, subclass, and unique characteristics. (Do not ask them to match bullets to guns. ) After the results are returned summaries should be sent to all participants for discussion Discrepancies should be discussed. The goal is to characterize how well different types of marks can be characterized. Follow on experiments could try different ammunition brands and types (of bullets-jacketed, hollow point etc.) Copyright Cliff Spiegelman 25

26 3. Error Rate Assessment Perform screening experiments from among the 50+ factors and variables to find out the most relevant and important ones affecting examiner performance. Physical factors such as gun type, and ammunition type, microscopes and lighting, etc. Social factors such as training and certification type These experiments will need complete buy in. Social psychologists should be consulted to help make tests representative of casework Copyright Cliff Spiegelman 26

27 Example of a screening experiment Copyright Cliff Spiegelman 27

28 Subjective Comparison Procedures Still Subject To Statistical Laws And Issues Multiple comparison issues Modeling errors (misspecified likelihoods particularly in tail regions) Need confidence statements Populations need to be clearly defined Data with annotation are needed Analyses need to grounded within the field of statistics

29 A D Composition

30 Composition A B C D By chaining, D becomes analytically indistinguishable from A

31 A Hypothetical: Example: Madrid Bombing

32 Inference Deductive inference: The fingerprints correspond therefore the suspect is the offender Not appropriate in most cases in forensic science Would involve explicit exclusion of all other sources than the suspected one

33 Inference Inductive inference If the suspect is the offender, then the fingerprints should correspond the fingerprints correspond therefore, it is possible that the suspect and the offender are the same person Appropriate to most cases

34 Intuitively depends on: Inductive inference

35 Inductive inference Intuitively depends on: Similarity between trace and control material Rarity of control material Background circumstances

36 Current Situation It is very difficult for an audience to appreciate what the scientist truly means, and how to use the conveyed information to reach a decision What and How are not considered separately in Forensic Science No real consensus on the What (not even the beginning of one in the US) No study on the How

37 There is no good way to report these conclusions Weight of evidence convey information regarding: Level of agreement between trace and control objects Level of rarity of the characteristics of the trace Potentially error rate(s) Potentially relevance, transfer and persistence It does not: Make assumptions on size of population of potential offenders Involve considering factors unrelated to the evidence But weight of evidence conveys the information in an obscure way, and the audience may not be able to readily use it in its decision-making process

38 There is no good way to report these conclusions Weight of evidence convey information regarding: Level of agreement between trace and control objects Level of rarity of the characteristics of the trace Potentially Thompson error rate(s) el al. (2013) Do Jurors Give Potentially relevance, Appropriate transfer Weight and persistence to Forensic It does not: Identification Evidence? Journal of Make assumptions Empirical on Legal size of Studies population 10(2) of potential offenders Involve considering factors unrelated to the evidence But weight of evidence conveys the information in an obscure way, and the audience may not be able to readily use it in its decision-making process

39 Psychology of Effective Communication Three main theories: Frequency theory Theorizes that human beings are more competent with counts than with probabilities because they have been exposed to them more across evolution. Cognitive experiential approach Originates from psychodynamics: different personality types, some relying more on numbers, some relying more on intuition. Intuition represents a lower level of development than numeracy. Fuzzy trace theory Originates from cognitive research: individuals rely on their gist (substance of information intuition ) and verbatim (exact representation of information - numeracy) to make decisions. Intuition represents a higher level of development than numeracy.

40 Psychology of Effective Communication Three main theories: Frequency theory Theorizes that human beings are more competent with counts Human than with relies probabilities on the because they have been exposed to them least more precise across gist evolution. Cognitive experiential approach Originates from psychodynamics: different representations personality types, some relying more on numbers, some necessary relying more to make on intuition. a Intuition represents a lower level of development decision than numeracy. Fuzzy trace theory Originates from cognitive research: individuals rely on their gist (substance of information intuition ) and verbatim (exact representation of information - numeracy) to make decisions. Intuition represents a higher level of development than numeracy.

41 Psychology of Effective Communication According to these theories, comprehension of information (verbal or numerical) and resulting actions/decisions are generally influenced by: Ability to mentally conceptualize the problem Format of the information Expectation Severity of the possible outcome Ability to retrieve knowledge/values from memory Past experience Specific context Cueing of relevant knowledge/values to consider Ability to apply reasoning processes Processing interferences

42 So right now what would the scientific method allow for testimony 1. In the opinion of the examiner the bullets and/or cartridge casings have marks consistent with being fired from the evidence weapon. The prosecutor and defense attorney should ask how many weapons might be consistent with this evidence. The examiner can answer a small percentage but should say that they do not know how small and that a few thousand weapons would be a small percentage.

43 So right now what would the scientific method allow for testimony 2. In the opinion of the examiner the evidence weapon cannot be eliminated as making the marks on the bullets and/or cartridge casings found at the crime scene. The prosecutor and defense attorney should ask how many weapons could there be that would not be eliminated as providing this evidence? The examiner can answer a small percentage but should say that they do not know how small and that a few thousand weapons would be a small percentage.

44 How can these statements be made stronger? By using statistically designed experiments Estimate the probability of misidentifying matches due to subclass characteristics. Use uncertainty intervals in testimony Estimate the probability of misidentifying a match from a weapon that does not share subclass characteristics. Use uncertainty intervals in testimony

45 Lack of statistical and mathematical input hurts us all If the wrong person is convicted in part or wholly due to overstated testimony or faulty forensic practice (say dry labbing) then the guilty person is free to commit other crimes, such as murder and rape. Of course long years on death row or worse for the wrongfully convicted are unintended consequences of the current forensic practice. This is a fixable problem.

46 Using IR images instead of visual light Copyright Cliff Spiegelman 46

47 More IR detail

48 More IR detail

49