Chapter Which of these are true and which are false? Explain why the false statements are wrong

Transcription

1 Chapter 8 1. Which of these are true and which are false? Explain why the false statements are wrong 1 a. By repeating a measurement many times, a researcher can improve the validity of the results. False it only helps with reliability. b. The average of 10 independent measurements will be more reliable than a single measurement made by the same instrument. True c. A measurement that is valid cannot be biased. False reliability, bias, and validity are three separate concepts and you can have any combination of good or bad properties of each. d. A larger sample size will help reduce bias in a sample survey. False it only helps with reliability. 2. A new type of thermometer is designed to measure temperatures in the extreme heat of an industrial furnace. Since pure iron melts at 2800 o F, the thermometer is tested five times by using it to measure the temperature when a sample of pure iron melts in the oven. In the five tests the thermometer registers o F, o F, o F, o F, and o F. In measuring these extreme temperatures, the new thermometer has the biggest problem with A. Bias B. Reliability C. Validity Pick one and explain. The measurements are very consistent (within 0.2oF of 2807oF) so reliability is excellent but they are systematically about 7oF too high. Validity would not be a concern as of is certainly an appropriate measurement scale to get at the issue of the thermometer s quality. 3. In 2008, the world s most accurate clock was the NIST-F1 built by the National Institute of Standards and Technology in Boulder Colorado. It is an atomic clock that is accurate to a tenbillionth of a second. Coordinated Universal Time is the official world time. The governing body that sets the Coordinated Universal Time uses an average of 250 different atomic clocks in 50 countries. a. A statistical reason for doing this instead of just using the time given by the NIST-F1 is that A. this is likely to be a more valid measure of the true time. B. this is likely to be a less biased measure of the true time. C. this is likely to improve the correlation between the times of the different clocks. D. this is likely to be a more reliable measure of the true time. b. In 2010 the NPL-CsF2 clock operated by the Nation Physical Laboratory of the United Kingdom was created to be about 10 times more accurate than the NIST-F1. Then, in 2013, a team of physicists

2 at the National Institute of Standards and Technology invented a new design that was another 100 times more accurate than the NPL-CsF2 and built two of the new clocks. Explain how these two new clocks can be used to create an even more accurate measure of time. Would these two new clocks be more help in establishing the level of bias or the reliability of the NPL-CsF2? They would provide an external standard by which you can judge the bias of the NPL-CsF2. 4. A reporter filing a story about a new convention center reports that the largest meeting room at the center is about 50 yards long and about 20 yards across. These measurements are based on his pacing off the room in each direction and reporting the number of paces he took. In making this report, he faces two obvious criticisms. Of course, pacing off a room is inaccurate because of the inconsistency in a person s stride. If he paced off the room a second time, he would almost surely get a different answer. Also, reporting the answer in yards is inaccurate because the reporter is rather short and his paces are not typically a full yard long. Discuss the key statistical properties of the reporter s measurements. Pacing off the room and calling it yards is a biased measurement since there are systematically more paces than there are yards in a measured length. Pacing is also an unreliable measurement since it is so inconsistent. However, pacing is reasonably valid since it does get at the issue of length. 5. In lab #4, students used a crude ruler to measure the width of the textbook. In doing so it was found that 1) when the ruler was used over and over again, the measurements came out dramatically different each time and 2) the measurements tended to be too low because the numbered markings on the ruler were not really an inch apart, although 3) at least the ruler did get at the issue of the size of the book, which is what was being investigated. From 1), 2), and 3) above we see that the measuring system was Issue (circle the correct choice in each row) Reliable or Not VeryReliable Valid or Not Valid Unbiased or Biased Reason (write 1, 2, or 3 as appropriate) The age of a pine tree was independently measured five times using a new electronic probe inserted in the tree s trunk. The measured values were 43 years old, 40 years old, 45 years old, 44 years old, and 41 years old. Later, this same tree was cut down and, by counting the growth rings, it was determined that the tree was really 34 years old. Does this new electronic device for measuring the age of trees have a greater problem with bias or with reliability? Explain. It has the bigger problem with bias since they are all within 3 years of 43 y old but are systematically too high (above 34) by more than that. 2

3 7. A teacher wishes to know if a new English curriculum will increase the creativity of her students in writing poetry. At the end of the unit, a sample of each student s poetry is given to a panel of five experts who rate the creativity of the poem on a 10-point scale. For each of the two statements below, explain whether it addresses the issue of measurement reliability or measurement validity. Statement I In reporting the results, the teacher claimed that the experts ratings demonstrated her students creativity. A critic of the new curriculum argued that the ratings were a poor way to measure creativity. This critic is arguing about a lack of validity by discuss how the measurement does a poor job of getting at the issue of creativity. Statement II The five experts seemed to rate each student nearly the same. For example, one student s poem got ratings of 8, 7, 8, 8, and 6 by the five experts. This statement says that the measurement is fairly reliable since repeating it gives fairly consistent values. 8. A report in The New England Medical Journal described a study of the effects of hormone therapy on middle-aged women. About 950 women took part in the study; half were selected randomly to receive the hormone therapy and the other half was given a placebo. After about a year, blood tests were conducted on each subject by a lab technician who was unaware of which group (treatment or placebo) the blood samples originated from. In presenting the results of the experiment, the authors reported that the women in the treatment group had experienced a statistically significant increase in HDL (the so-called good cholesterol) and a significant decrease in LDL (the so-called bad cholesterol) when compared with the control group. a. Describe what is meant by the words statistically significant in the paragraph above. It means that the differences between the hormone and placebo groups was unlikely to have happened by chance. b. A newspaper report said the experiment had shown that hormone therapy is effective in reducing the women s risk of heart attack. This is not a justified conclusion because: A. the design of the experiment was seriously flawed. B. the cholesterol measurements were not a reliable measure of the risk of heart attack. C. the cholesterol measurements may not be a valid measure of the risk of heart attack. D. there is no reason. The conclusion is justified. Pick one and explain briefly. While high levels of bad cholesterol may be associated with heart attacks it is not the same thing. If the issue is risk of heart attack then cholesterol measurements may have a serious issue with validity. 3

4 9. A team of engineers developed a new device to measure the speed of lava flows in volcanic areas. The device uses an infrared laser to measure the speed. The researchers set up a test in Hawaii to measure a lava flow that is moving at a known speed of 20 feet per minute. In a test of 10 samples, the engineers find the following measures: 25, 12, 19, 20, 18, 22, 28, 15, 21, and 20. Does this device have a bigger problem with bias or reliability? Explain how you know. It has a bigger problem with reliability. While the true value of 20 is near the center of the measurements (it is the median), the repeated measurements bounce around by as much as 8 feet per minute. 10. A common measurement of the lifetime smoking habits of subjects in observational studies dealing with heart disease is the number called Pack Years. This number is computed as: 4 (lifetime average number of packs smoked per day) x (number of years smoked) A critic gives three reasons why Pack Years is not a perfect measurement: Reason I Often, when asked multiple times, subjects will give you different answers, about how much they smoked at various times in their lives. Reason II The average number of cigarettes smoked per day is not the only facet of a subject s smoking habits that is of interest. For example, some researchers believe that it is worse to smoke half pack a day for a year followed by a pack and a half a day for another year than to smoke a pack a day for two years. Reason III Subjects often lie about how much they have smoked, almost always tending to downplay the true amount. a. Which of these criticisms argue about the validity of the Pack Years number? Reason II it talks about whether the pack years measurement gets at the issue of lifetime smoking habits effect on heart disease. b. Which one argues about the reliability? Reason I it talks about the consistency of the measurement when repeated. c. Which one argues about bias in the measurement? Reason III it says that the measurement is systematically too low. 11. The diameter of the moon is independently measured four times by a process that is free of bias. The individual measurements come out 2157, 2166, 2162, and 2155 miles, which average out to 2160 miles. One more measurement is about to be taken using the same process. When compared with the estimate of 2160 miles, you would expect this next measurement to be: A. more accurate as a measure of the true diameter of the moon. B. just as accurate at measuring the true diameter of the moon. C. less accurate as a measure of the true diameter of the moon. Pick one and explain. The average of four measurements will be more reliable than a single measurement and hence more accurate since all of the measurements are unbiased.

5 12. Two chemists make measurements of the amount of lead in a sample of the city s water. The chemists each repeat their measurements 20 times. In reviewing the results, the public works manager finds that the measurements made by the first chemist had a much greater variability than the measures made by the second, but the averages for the two were the same. This implies: A. The first chemist has made more reliable measures than the second. B. The first chemist has made more biased measures than the second. C. The first chemist has made less reliable measures than the second. D. Both B and C. 13. Trucks are weighed at a truck scale to establish the amount owed in road taxes. Someone complains that the weighing procedure has three problems. Problem I Sometimes the driver is sitting in the truck when it is weighed. Problem II When the same truck is independently weighed more than once, the truck scale will give different values. Problem III When the legislature established the road tax, they intended to tax according to the value of the goods being shipped, not according to the weight. Explain how each of these three problems relate to the properties of the measurement at hand. Problem I talks about bias (systematically too high because the driver s weight would be added. Problem II talks about reliability (consistency). Problem II talks about validity (getting at a different issue than desired). 14. Two investigators measure the depth of an oceanic trench. The first one makes 20 independent measurements and reports the average of the values obtained; the second person reports the result of a single measurement using the same process. a. Which investigator will have more reliability in the number reported, or will they both have about the same reliability? The first investigator will have greater reliability since the average is less variable than a single measurement. b. Which investigator will have more bias in the number reported, or will they both have about the same bias? The two investigators will have the same level of bias since they are both using the same process. 15. True or False: If a measurement is valid then it must have high reliability. False these are separate properties you can easily have one without the other. 16. A watch is running exactly 5 minutes fast all day long. True or False: in measuring the time, this watch is reliable, valid, and biased. True 5

6 17. Apgar scores are a measurement of an infant s overall health taken a few minutes after birth. The score ranges from 0 (dead) to 10 (perfect health) and are based on tests of the baby s heart and breathing rate, muscle tone, etc... A critic gives three reasons why the Apgar score isn t a perfect measurement: Reason I There are many important facets of health that aren t measured by the score. Reason II A doctor's rating may be affected by being present at the birth, often giving unwarranted low values to babies whose birth was difficult. Reason III Two different doctors may give different Apgar scores, even when measuring the same baby at the same time. a. Which of these criticisms argue about the validity of the Apgar score? Reason I it s a reason why the measurements may not get at the issue at hand. b. Which one argues about the reliability? Reason III it s a reason why the measurements may show some variation. c. Which one argues about bias in the measurement? Reason II it s a reason why the measurement is systematically too low. Explain your choices. d. Suppose two doctors both judge an infant s health using the Apgar system and the average of their two values is taken as the official Apgar score. Explain why this would be beneficial in measuring the infant s health? It will help with reliability since the average is less variable than a single measurement. 18. A nutritionist for the FDA develops a new machine that quickly measures the amount of fat in foods. The nutritionist s supervisor examines this machine and has the following comments: When I tested the machine on a standard sample that we know to have 25 mg of fat we found it always gave values that were too high, although they only varied by plus or minus 1 mg. Also, the machine measures caloric content rather than true fat content. Discuss the statistical measurement properties of the new machine. The supervisor is saying that the new machine gives biased measurements (systematically too high), reliable measurements (very consistent) and have an issue with validity (not really getting at the issue of fat content). 19. A new hand-held instrument has been developed to measure carbon monoxide levels in the air. The device is tested 5 times independently on a sample of air that was made to contain 10 ppm (parts per million) of carbon monoxide. In measuring this sample, the device records readings of 9.56, 9.53, 9.54, 9.51, and 9.55 ppm. We can see from this data that the measurement process has A. more problems with bias than reliability B. more problems with reliability than bias C. equally serious bias and reliability problems D. not enough information to tell Pick one and explain. Systematically too low by about 0.46 ppm but fairly consistent 6

7 20. A researcher has devised a new questionnaire that attempts to measure a person s stress level. In doing so it was found that 1) the results from the questionnaire were much less consistent than the results from a longer version that had been developed earlier. 2) people often responded to the questions in a way that tended to over-emphasize how much stress they were under. 3) at least the questionnaire did seem to get at the issue of the amount of stress a person was under. From 1), 2), and 3) above we have reasons that the questionnaire was Issue (circle the correct choice in each row) Reason (write 1, 2, or 3 as appropriate) Reliable or Not Very Reliable 1 Valid or Not Valid 3 Biased or Not Biased 2 EESEE Exercises The following exercises make use of stories in the Electronic Encyclopedia of Statistics Examples and Exercises, or EESEE (pronounced ee-zee). EESEE is included in the StatsPortal materials that accompany the textbook under the Resources tab. You will find the specific stories listed alphabetically by title. 21. EESEE Story Hunting in the Rain Forest. Researchers Dwyer and Minnegal from the University of Queensland, Australia studied the hunting habits of the Kubo People who live in the Gwaimasi village on the west bank of the Strickland River in the tropical rain forest of New Guinea. They noted when each resident left and returned to the village and when any groups of residents returned from a hunting trip, they were asked who should be credited with each capture. When more than one resident was involved in the capture of a single animal, credit was split evenly amongst them. Data on the number of kilograms of meat captured for each type of animal by each resident was recorded for a full year. a. Complete question #1 from the Hunting in the Rain Forest EESEE story. a) The "Total" variable gets at this issue directly although in presenting these values it is important to note that some of the males were only transient residents of the village and others were children. b) To measure efficiency we should standardize the total catch by the number of opportunities. Since we don't have this information directly, a reasonable substitute would be to divide the total spear fishing catch by the "Nights" variable to get a catch 7

8 per day measure (this is essentially what Dwyer and Minnegal do in their article). In presenting this information it should be mentioned that this measures efficiency only if each male makes attempts at spear fishing at a comparable pace. c) To measure the weight of fish captured, we need to total the amounts captured by spear, by hook, and by other means. To report a typical value we might use the median of these totals. In presenting this value as typical it should be pointed out that the list included children with zero or negligible values. d) Summing the values in the "Nights" column and dividing by 398 (the # of nights in the period between 9/15/86 and 10/18/87) would give an average. In presenting this average, it should be noted that the men may often travel away from the village together (e.g. the fathers with their sons) so that this average may not be typical. e) The efficiency at spear and hook-and-line fishing might each be measured using catch per day values (see Part b). To measure the strength of the association between these two we might use Pearson's correlation coefficient if a linear association is expected or the rank correlation if only a monotone association is expected (this is what Dwyer and Minnegal do in their article). The caveat mentioned in Part b still applies here. b. For each measure that you describe in part a, explain whether you feel it is most likely to have problems with reliability, bias, or validity. The discussions above are primarily about the validity of the measurements since they describe ways that the measures might not get at the issues involved. 8