Chapter 1: Alternative Forced Choice Methods

Transcription

1 Chapter 1: Alternative Forced Choice Methods Section 1.1 Birdnapping Lewiston man confounded by stolen parrot art Lewiston, Minn Jim Schloegel stood under the shade of a giant bird cage and raised both arms in confusion. His giant plywood parrots hung above him, waving slightly against their tethers. Last week, someone purloined one of his brightly painted works of art. I m more puzzled as to why someone would take it, Schloegal said. Why? In an effort to convince the local police of your innocence, they have been asked to take the following quiz. You must answer all questions. 1. T F This residence is about 1 mile from Lewiston. 2. T F A lock was broken to enter the bird cage. 3. T F The artist s signature was on tail of the stolen bird. 4. T F This residence is on gravel road. 5. T F The most prominent color on the stolen bird was green. 6. T F The stolen bird had a yellow beak. 7. T F The date the bird was made was painted on the tail. 8. T F There was a horse in the pasture next to the bird cage. 9. T F The stolen bird had a red eye. 10. T F There was a car parked in front of the shed at the scene. 11. T F The skies were clear the evening the bird was stolen. 12. T F There was a light on in the house at the scene. 13. T F The bird cage had a cement floor. 14. T F Wire was used to suspend the birds in the bird cage. 15. T F A car drove by this residence as the bird was being stolen. 16. T F When the bird was removed, part of its beak was damaged and left on the floor of the bird cage. 17. T F One of the two get away vehicles was a pickup. 18. T F Three individuals entered the bird cage. 19. T F The hinges on the door to the bird cage squeaked when opened. 20. T F Flowers were planted around the perimeter of the bird cage. Number Correct: 1

2 Consider the following number line for the number of correct responses from the above quiz. Questions 1. In the context of the birdnapping quiz, what does it mean if an individual answered 20 out of 20 correct? Explain. 2. If an individual had no knowledge of this crime, what would be their expected or anticipated score on this quiz? 3. In the context of the birdnapping quiz, what does it mean if an individual answered 0 out of 20 correct? Explain. Consider an individual who has knowledge of this crime, but in an attempt to elude police is intentionally answering most questions incorrectly. Such an individual is likely not to answer all the questions incorrectly as this would draw attention to them. Alternative forced choice studies have shown that it is difficult for such an individual to follow the patterns of people with no knowledge of the crime. 4. In your opinion, what would be a reasonable cutoff value for when you start to believe an individual is trying to intentionally give wrong answers (i.e. throw the police off )? 5. Ask some of your neighbors at what point they would become convinced that an individual is trying to throw the police off? Neighbor 1: Neighbor 3: Neighbor 2: Neighbor 4: 6. What potential issues arise when different people use different values for the point at which they become convinced an individual is intentionally giving wrong answers? 2

3 An important statistical question is how to determine whether an individual s score on the 20 question true false quiz is surprising under the assumption that they are simply guessing on each question. To answer this question, we will simulate the process of guessing on 20 true false questions several times. Each time we simulate the process, we ll keep track of how many questions an individual answered correctly (note that you could also keep track of the number of incorrect answers). Once we ve repeated this process several times, we ll have a pretty good sense for what outcomes would be very surprising, or somewhat surprising, or not so surprising under the situation that an individual is really guessing. To simulate this process, we will flip a fair coin for each of the 20 questions. To be consistent, let heads represent a correct response and tails an incorrect response. An individual answers correctly An individual answers incorrectly Question (Coin) Outcome Tally the total number of correct answers: Plot your number of correct answers on the number line below. Add to this plot, the results from your classmates. 3

4 Questions Answer the following using the results from the entire class. 7. How many simulations (i.e. dots) are represented in the above plot? 8. If an individual is really guessing, what outcomes are likely to occur? 9. If an individual is really guessing, what outcomes are not very likely to occur? 10. Using the results from our class simulation, how many of the 20 questions would an individual have to answer correctly in order for you to be convinced they were intentionally giving the wrong answers? 11. How does your cutoff value here (i.e. the one based on the fair coin simulations) differ from the cutoff value given earlier (i.e. see Question 4)? 12. Which cutoff would be statistically better to use? Explain. 4

5 Why does flipping coins work? Flipping a fair coin repeatedly has the following characteristics. Explain how each relates to an individual guessing on the 20 question birdnapping quiz. Fair Coin Characteristics Relationship to 20 Question Quiz Two outcomes: heads and tails 50 / 50 chance for each outcome Fair coin will be flipped a specified number of times (i.e. 20 times here) Answer the following questions regarding the potential outcomes from fair coins and this 20 question quiz. Using Coins 1a. If you toss a coin 20 times, how many coins would you expect to land on heads? Using The 20 Question Quiz 1b. If an individual is truly guessing on each question, how many questions would you expect this individual to answer correctly? 2a. A classmate tosses a coin 20 times, and the coin lands on heads 8 times. They claim their coin is biased against heads. What is wrong with their reasoning? 2b. An individual answers only 8 out of 20 questions correctly on the quiz. The police believe that since this was less than the expected number of correct answers, this individual must be intentionally answering incorrectly in order to mislead them. What is wrong with this reasoning? 3a. A classmate tosses a coin 20 times, and the coin never lands on heads. They claim their coin is biased against heads. Do you agree with their reasoning? 3b. An individual answers 0 out of 20 questions correctly. The police believe that the suspect must be intentionally answering incorrectly. Do you agree with their reasoning? 5

6 A key question is how to determine whether an individual s score on the 20 question true false quiz is surprising under the assumption that they are simply guessing on each question. To answer this question, we will simulate the process of guessing on 20 true false questions several times. Each time we simulate the process, we ll keep track of how many questions the suspect answered correctly (note that you could also keep track of the number of incorrect answers). Once we ve repeated this process several times, we ll have a pretty good sense for what outcomes would be very surprising, or somewhat surprising, or not so surprising under the situation that an individual is really guessing. Various technologies can be used for these simulations. For example, the web page has been specifically set up for 2 alterative forced choice simulations. For this simulation, you should specify the labels for the two outcomes and specify the number of repeated trials (i.e. 20 for the birdnapping quiz). Click Run > to obtain the outcome from a single simulation. Use the simulation on this web page to obtain 20 outcomes. For each simulation, record the number of correct and plot the result on the number line below. 6

7 Questions: 13. How many simulations (i.e. dots) are represented in the above plot? 14. What outcomes would be very surprising to observe if the suspect is really guessing? 15. What outcomes would be NOT very surprising to observe if the suspect is really guessing? 16. Using the results from your 20 simulation, how many of the 20 questions would an individual have to answer correctly in order for you to be convinced they were intentionally giving the wrong answers? 17. How does your cutoff based on this simulation (i.e. question 11) compare to your cutoff without the simulation (i.e. question 5)? 18. Which cutoff do you have more confidence in? Explain. 19. Again, ask some of your neighbors at what point they would become convinced that an individual is trying to throw the police off? Neighbor 1: Neighbor 3: Neighbor 2: Neighbor 4: 20. How do these cutoff values compare to one s given earlier (see Question 6)? Are these more consistent with each other? Why is consistency a good thing? 7

8 Consider the following graph from 20 trials. 1. Is your graph the same as the one presented above? Should it be exactly the same? Explain why or why not? 2. Your friend makes the following statement regarding the graph above. If the Number of Correct Answers is around 2, 3, 4, or 5; then I believe an individual is intentionally giving wrong answers. Your simulation did not produce any values in this region, so you must have done something wrong in the setup of your simulation. Do you agree or disagree with this statement? Explain. 3. Your friend makes the following statement regarding the graph above. Ten is the expected number and should be the most common outcome. However, this simulation resulted in eleven being the most common; thus, you must have done something wrong in the setup of your simulation. Do you agree or disagree with this statement? Explain. 8

9 Section 1.2 Using Technologies to Obtain Reference Distributions Using Tinkerplots to Repeatedly Flip Coins To save time and to gather more data quickly, we will use a software package called Tinkerplots to simulate tossing 20 coins many times. We will then use the data we generate from this simulation to help understand likely outcomes for the number of heads (i.e. the likely outcomes for someone who is guessing). Setting up the Simulation in Tinkerplots Open Tinkerkplots on your computer. Drag a new Sampler from the tool shelf into your blank document, as shown below. The default sampler is called a Mixer sampler which simply a hat that contains three objects (two objects labeled as a and one object labeled as b. Tinkerplots gives you options on the type of sampler to use for a particular situation. To begin, we will use the Spinner sampler. Mixer Sampler (Default) Spinner Sampler 9

10 The default labeling for the outcomes on the Spinner are a and b. Relabel the outcomes as Heads and Tails. Fair coins have a 50% chance landing on Heads and 50% landing on Tails. Thus, we need to change the proportion for Heads and proportion for Tails. This can be done by selecting Show Proportion the dropdown menu to the lower left of the Spinner sampler. Your Spinner sampler should look as follows. Two additional changes need to be made to the Spinner sampler. Change the Draw value from 2 to 1 and the Repeat value to 20. This will simulate tossing a single coin 20 times. 10

11 Your completed Spinner for tossing a single coin 20 times should look as follows. Running the Simulation 4. Click the Run button in the upper left corner. A table of the outcomes for 20 fair coin tosses will be generated. What was the outcome from the first coin toss? How about the 10 th coin? 5. How many Heads did you get out of the 20 coin tosses? Next, let s use Tinkerplots to plot the 20 outcomes from your first trial of the simulation. This will allow us to more easily count the number of Heads. Start by selecting the variable name of the variable to be plotted, Coin in my case and then select Plot. 11

12 Drag a new Plot into an open space. The original plot does not separate the Heads from the Tails. You can separate the dots by dragging a dot to the left or right. Plot of Coins Separate the points Tinkerplots has the ability to automatically count the number of Heads and Tails on your plot. Select the N icon from the menu bar. Click N icon to place Counts on your graph. Graph with counts shown. 12

13 Collecting the Results from Many Trials Next, we will run many more trials and use Tinkerplots to collect the number of Heads obtained each time. To record the total number of heads for each trial of 20 coin tosses, right click on the value for the number of Heads in your plot and select Collect Statistic. To run additional simulations, say 100 total, put 99 in the Collect box and click Collect. This will add 99 additional trials to our initial simulation. 99 Questions 6. How many heads were obtained on the 47 th trial? 7. What is the least number of heads in these 100 trails? How about most? 13

14 Plotting the Results from the 100 simulations Although we can see the individual results in the table shown above, this is not necessarily a good way to organize the results. A better way to organize these results is to plot them. Plot the results from your 100 trials as shown below. Drag a new Plot onto your workspace. Next, highlight the variable name from your collection in which the outcomes from the 100 trials are stored (count_coin_heads) and drag this onto the bottom of the new plot. In order to count the number of 8 s, 9 s, etc. from each of trail, drag a point all the way to the right so that the plot displays individual columns for each outcome. The vertical Stack button can be used to neatly stack the points. Also, the size of the plotting symbol can be reduced with the slider on the bottom of the plot. 14

15 Dotplot showing the outcomes from the 100 trials of this simulation. The plot above does not display all possible values from flipping 20 coins. This plot should start at 0 and go up to 20. Double click on the smallest axis label, in the Edit Axis box, put 0 in for Axis starts at and set the Bin width to 1. This is shown here. 15

16 Next, set the largest axis value to 20. Finally, to obtain a count for each value, simple select the N button on the menu bar. Sketch your plot on the graphic below. 16

17 Questions 8. How many coins were flipped for a single run or trial in this simulation? 9. What does each dot on the graph represent? 10. If an individual is really guessing, what outcomes would be surprising to observe? 11. If an individual is really guessing, what outcomes would not be surprising to observe? That is, what are common values we d expect to see if an individual is really guessing? 12. What is the smallest value from one trial of your simulation? a. Is it possible to obtain a value smaller than this if more simulation were done? b. If one additional simulation is done, is this outcome likely to be smaller, larger, or about the same as the previous smallest value? Explain. 13. A statistician might argue that a plot based on 100 trials is better than a plot based on 20 trials. Do you agree? Explain why or why not. 17

18 Consider the following graphs, one based on 100 trails and the other based on 1000 trials. Graph A: 100 trials Graph B: 1000 trials Questions 14. What are the similarities between the two graphs? Discuss any differences as well. 15. What would you consider an unusual outcome (i.e. outlier) to be in Graph A? How about Graph B? 16. A statistician might argue that the determination of what constitutes an outlier should not necessarily be influenced by the number of trials. For example, a statistician might say that a value less than 5 would be considered an outlier in either graph. Do you agree? Explain why or why not. 18

19 Section 1.3 Applications of Two Alternative Forced Choice in Forensic Science Consider the case studies presented in Hall and Thompson (2007). Two alternative forced choice methods (or Explicit Forced Choice as referred to by the authors) were used to evaluate the truthfulness or lack thereof for each of the case studies presented in this article. In the first two case studies, 24 questions were used. These questions were binary in format which means each question had two possible outcomes. References: Harold V. Hall and Jane Thompson. Explicit Alternative Testing: Applications of the Binominal Probability Distribution to Clinical Forensic Evaluations. The Forensic Examiner, Spring Example 1.3.1: Consider the suspect MT case study presented in the article. For the forensic evaluation of this individual a total of 24 binary questions were used. The context of these questions was centered around this crime and included several characteristics of the victim that only the perpetrator would know. Questions 1. The highest possible score by an individual given this sequence of questions is 24. In the context of this example, what would it mean if an individual correctly answered all 24 questions? 2. The lowest possible score by an individual given this sequence of questions is 0. In the context of this example, what would it mean if an individual correctly answered none of the questions? 3. If a person had no knowledge of crime and/or victim, what would be their anticipated score? How did you make this determination? 19

20 A simulation will be setup in Tinkerplots to mimic the likely outcomes for an individual given this sequence of 24 questions who has no knowledge of this crime. A total of 100 trials were obtained by Tinkerplots and the resulting outcomes are presented here. Questions 4. How many dots are on this plot? 5. What does each dot on the graph represent? 20

21 6. Under which situation were these dots generated? i. These dots were generated under the situation of no knowledge of crime ii. iii. These dots were generated under the situation of some knowledge of the crime These dots were generated under the situation of complete knowledge of the crime Explain your reasoning. 1. Using the outcomes from the above simulation of 100 trials, how many of the 24 questions would an individual have to answer correctly in order for you to be convinced they were intentionally giving the wrong answers? Sketch this cutoff value on the number line below. Explain how you obtained this cutoff value. 2. This article states that suspect MT answered 13 of the 24 question correctly. Is 13 beyond your cutoff value for when you start to believe an individual is intentionally giving the wrong answers? Explain. 3. Does an outcome of 13 provide evidence to suggest that suspect MT has knowledge of this crime and/or victim? Explain. 21

22 Example 1.3.3: For this example, consider the case study presented in the article titled A Forced Choice Technique to Evaluate Deafness in the Hysterical or Malingering Patient by Loren Pankratz, Stephan Fausti and Steve Peed. A two alternative forced choice method was used to evaluate a patient s alleged hearing loss. Reference: Pankratz, L., Fausti, S., and Peed, S. (1975). A Forced Choice Technique to Evaluate Deafness in the Hysterical or Malingering Patient. Journal of Consulting and Clinical Psychology, Vol. 43, No. 3, pp The methodology used in this case study is presented below. In addition, a mock up of the simulation is available on line. Methods section from article. Mock up of Simulation To anticipate the range of likely outcomes for a deaf person in your simulation, mute your computer speakers. The article above mentions 100 trials was used to evaluate this patient. Each trial consisted of a single tone being paired with either a red or blue light (i.e. two choices). 22

23 Questions 1. If 100 trials were used, what would be the anticipated score for a hearing person? 2. If 100 trials were used, what would be the anticipated score for a deaf person? 3. What would it mean if a person got 0 out of 100 correct? Explain. 4. What would you use as a cutoff for when you start to believe that somebody is being untruthful in their ability to hear the tone? 5. Ask some of your neighbors at what point they would become convinced than an individual would be misleading in regards to their ability to hear the tone. Neighbor 1: Neighbor 3: Neighbor 2: Neighbor 4: 6. How does your cutoff value compare to your neighbors? 23

24 In order to evaluate this patient s alleged hearing loss, we will compare the outcome from his 100 trials (i.e. 36) to the likely outcomes from of a deaf person. Such a simulation can be done easily on a computer when you mute the computer speakers. Mute your computer speakers and run 20 trials of the simulation using the link below. Guess: Red or Blue Outcome: Correct or Incorrect Guess: Red or Blue Outcome: Correct or Incorrect Questions 7. How many Correct did you get in your 20 trials? Next, ask 4 neighbors for their number Correct as well. This will give us a total of 100 outcomes, which represents 1 trial of this experiment. You: Neighbor #1: Neighbor #2: Neighbor #3: Neighbor #4: Total 24

25 Plot the number correct from these 100 outcomes on the number line below. This represents one trial of the experiment. Questions 8. Is the outcome from this single trial close to the anticipated outcome for a deaf person? Explain. 9. Your friend makes the following false statement. You and your neighbors must have done something wrong in your first trial because the outcome from this single trial is supposed to be an outlier when compared against the anticipated outcomes from a deaf person. Explain why this statement is false. 10. Explain why a single trial is not enough to evaluate whether or not the outcome (36 out of 100 correct) from the patient in the case study is unusual when compared against the anticipated outcomes from a deaf person. Setup and run a Tinkerplots simulation to obtain 100 trials of this simulation. Plot the outcomes from your 100 trials on the plot below. 11. Using the 100 trials on the above plot, do you believe the patient in this case study is being deceitful in his ability to hear the tone? Explain. 25

26 Section 1.4: Additional Applications of Two Alternative Forced Choice Models Example 1.4.1: Ear Infections (Source: Rosner) A common symptom of otitis media (ear infection) in young children is the prolonged presence of fluid in the middle ear. The hypothesis has been proposed that babies who are breast fed for at least 1 month may build up some immunity against the effects of the condition. A small study of 24 pairs of babies is set up, where the babies are matched on a one to one basis according to age, sex, socioeconomic status, and type of medications taken. One member of the matched pair is a breast fed baby and the other was bottle fed. The primary outcome measurement recorded in this study was the duration (in days) of fluid in the middle ear after the first episode of otitis media. The results from the 24 pairs are below. Of interest is to make comparisons between the breast fed and bottle fed babies. These comparisons should be done within each pair of babies because of the auxiliary factors that were considered in this study. Who did better in head to head comparisons? Definition Response Variable: The primary outcome or measurement of interest in an analysis. o Also known as: Dependent Variable or Y Variable 26

27 Count the number of times breast fed and bottle fed babies did better and complete the following table. Outcome Number of Pairs Bottle fed did better Breast fed did better Tie Total 24 Question 1. Pair #8 is a tie, what does this mean in the context of this problem? Does Pair #8 provide evidence for bottle fed doing better, breast fed doing better, or neither? Explain. Consider the following pamphlet on Ear Infections in Children from the Department of Health from the State of New York. Link to Pamphlet: 27

28 Do a Google search and identify other factors (i.e. called Risk Factors) that are thought to influence the likelihood of a child getting an ear infection. Mayo Clinic Link: infections/ds00303/dsection=risk factors Question 2. What are some of these risk factors? Discuss their potential influence. Setting Up an Experiment of this Type Consider the following mock situation. A researcher has obtained 48 volunteers for their study. They have obtained important demographic variables for each of these 48 study participants and ask you to determine how to best match up these two sets of participants so that comparisons will be done most fairly. Goal: Propose a matching strategy for this study using the demographic information provided below. (This data is provided on course web site.) Question 3. Discuss the process used for matching these pairs of babies for this mock experiment. 28

29 Consider the following mock study participants. Notice the ages for the bottle fed babies is considerably higher than the ages for the breast fed babies. Questions 4. Suppose Age is known to influence the occurrence of ear infections. Explain why the differences in the ages between the two groups hinders our ability to compare these two groups. Definition Confounding Variable: A variable that cannot be delineated from another variable when attempting to establish a relationship with the response variable. 29

30 In all our analyses thus far, we have been restricted to only two outcomes. Recall, for this example we have three outcomes: 1) breast fed better, bottle fed better, and one tie; as a result, when we construct our spinner in Tinkerplots, we will not include the outcome from the tie. Questions 5. If the tie is removed, how many pairs do we have in our sample? 6. If there is no difference in the duration of ear infection between breast fed and bottle fed, for how many pairs should the bottle fed babies do better than the breast fed babies? Set up a simulation in Tinkerplots to investigate the situation for which there is no difference between the bottle fed and breast fed babies. In your simulation, you should track the number of breast fed pairs. Questions 7. What would it mean in the context of this problem, if the outcome from our sample was at the smallest possible value on our graph? 8. What would it mean in the context of this problem, if the outcome from our sample was at the largest possible value on our graph? 30

31 9. Is the outcome from our sample (i.e. 16 pairs for which breast fed doing better) an outlier? Discuss. 10. For this example, we will have two cutoff values. The reason we have two cutoff values is because the original question asked if there was a simply a difference (i.e. no preference to bottle fed or breast fed was given). Upper cutoff value: Lower cutoff value: 11. Does the observed outcome from our sample provide enough statistical evidence to suggest breast or bottle fed babies have a lower duration of fluid in their inner ear? Explain. Example 1.4.2: Gender Discrimination This fictitious example involves an evaluation of possible discrimination against female employees. Suppose a large supermarket chain occasionally selects employees to receive management training. A group of female employees has claimed that they are less likely than male employees of similar qualifications to be chosen for this training. The large employee pool that can be tapped for management training is 60% female and 40% male; however, since the management program began, 9 of the 20 employees chosen for management training were female (only 45%). Question of Interest: Is there evidence of gender discrimination for those chosen for management training? 31

32 Setting up the Simulation Study To investigate this research question, we will carry out a simulation in Tinkerplots 2. Once again, note that you will have to revise a few elements of the simulation that relate to the following questions: Questions 1. What are the two possible outcomes for each trial? 2. What is the chance or probability for each outcome, given that there is no discrimination? 3. How many employees were selected for management training? Use the answer to the above questions to setup the appropriate spinner in Tinkerplots. 32

33 For this problem, it is necessary to change the chance or probability on the spinner to match the situation presented here. To specify a proportion different than 50/50, select Show Proportion from the drop down menu on the lower left of the spinner. This is shown here. Next, change the proportion to the appropriate values. The following shows the outcome from one trial. Questions 4. In this study 20 individuals were selected for management training. The anticipated number of females is *not* 10. It actually larger than 10? What is the anticipated number of females? Explain. 5. Propose a formula for determining the anticipated number of females. Use your formula to determine the anticipated number of females if 50 individuals were selected for management training. 33

34 The following plot shows the outcomes from 1000 trials. Questions 7. Under which situation were these dots generated? iv. These dots were generated under the situation of possible discrimination against women and that this discrimination was intentional. v. These dots were generated under the situation of possible discrimination against women, but believe the discrimination was unintentional. vi. These dots were generated under the situation of no discrimination against women. Explain your reasoning. 8. What does each dot on the graph represent? 9. Using the outcomes from the above simulation of 1000 trials, how many women (out of 20) would you have to see in order to say discrimination is likely occurring? Sketch this cutoff value on the number line below. Explain how you obtained this cutoff value. 10. The outcome from this study had 9 women out of 20 selected for management training? Does this outcome provide evidence to suggest discrimination against women is occurring? Explain. 34