How Confident Are Yo u?

Mathematics: Modeling Our World Unit 7: IMPERFECT TESTING A S S E S S M E N T PROBLEM A7.1 How Confident Are Yo u? A7.1 page 1 of 2 A battery manufacturer knows that a certain percentage of the batteries p roduced are defective. This percentage depends on the quality of the p roduction process. If all machines are working as well as possible, only 2% of the batteries produced are defective. Due to the lack of machine maintenance, this percentage can gro w. If more than 4% of the batteries are defective, p roduction is stopped and the machines are adjusted. The problem is that the manufacturer doesn t know the exact percentage of defective batteries. To make an estimate of this percentage, he takes a sample of 40 batteries and tests all of them. Two out of these 40 are defective. 1. Suppose you were the manufacturer. Based on this sample, what would you conclude about the percentage of defective batteries in the total produced? 2. Why is it dangerous to draw conclusions based on this one sample? Instead of drawing conclusions on just one sample of 40 batteries, the m a n u f a c t u rer decides to repeat this sampling 50 times, using 40 new batteries in each sample. All batteries are taken randomly from the daily pro d u c t i o n. The results are listed in Figure 1. Number of defective batteries 0 1 2 3 4 5 6 Number of samples 20 19 7 3 1 0 0 Figure 1. Defective batteries in 50 samples of size 40. 3. What is your best estimate for the percentage of defective batteries in a sample of size 40 based on these 50 samples? Explain your answer. 4. Estimate the percentage of defective batteries in the total production. How confident are you that your estimate is exact? 6 4 5

A S S E S S M E N T Unit 7: IMPERFECT TESTING Mathematics: Modeling Our World A7.1 page 2 of 2 The data in Figure 1 were created by running a simulation program similar to the Whitesocks p rogram (Handout H7.4). Using this program, data from 50 samples of 40 batteries each were simulated, and each battery had a 2% chance of being defective. 5. Why is there no guarantee that the estimate based on the samples will be exactly 2%? Figure 2 shows the results of 50 samples of 40 batteries each. Figure 3 shows the results of 50 samples of 100 batteries each. Number of defective batteries 0 1 2 3 4 5 Number of samples 6 13 15 10 4 2 Figure 2. Defective batteries in 50 samples of size 40. Number of defective batteries 0 1 2 3 4 5 6 7 8 9 Number of samples 0 3 7 5 10 7 5 8 2 3 Figure 3. Defective batteries in 50 samples of size 100. 6. For each table, estimate the percentage of defective batteries in the total production. 7. Which one of your two estimates do you think is the better estimate for the percentage of defective batteries? Why? 6 4 6

Mathematics: Modeling Our World Unit 7: IMPERFECT TESTING A S S E S S M E N T PROBLEM A7.2 How to Quit Smoking A7.2 page 1 of 2 An analysis in the Journal of the American Medical Association stated that nicotine patches, used by over four million Americans since they were first marketed in 1991, were at least twice as effective in helping people to quit smoking as placebo (fake) patches in 17 studies involving 5098 people. After a four-week treatment period, 27% of the nicotine-patch wearers were cigarette-free, versus 13% of placebo-patch users. Six months after treatment, 22% of the nicotine-patch users were still not smoking, compared to only 9% of the placebo users. Studies are done comparing the effects of fake patches and nicotine patches to test whether nicotine patches really do help people quit smoking. The analysis above states that nicotine patches are at least twice as effective as fake patches. In studies involving 5098 people, ultimately 22% of the nicotinepatch users stopped smoking, compared to 9% of the fake-patch users. The article doesn t state, however, what number of people got nicotine patches or fake patches. In the following questions, you will investigate some different possibilities. Assume that the percentages mentioned above are correct. 1. Suppose that in one study involving 4000 people, exactly the same number of people stopped smoking after using nicotine patches as after using fake patches. a) Without doing any calculations, describe a condition that would allow this to happen. b) What percentage of these people had the nicotine patches, and what percentage had the fake patches? 6 4 7

A S S E S S M E N T Unit 7: IMPERFECT TESTING Mathematics: Modeling Our World A7.2 page 2 of 2 c) Under the conditions of this problem, estimate the number of people who used the nicotine patches and the number of people who used the fake patches. 2. Would it be possible for more people to quit after using fake patches than after using nicotine patches? Explain why (or why not). 3. Explain why the study concluded that nicotine patches are at least twice as effective as fake patches. 6 4 8

Mathematics: Modeling Our World Unit 7: IMPERFECT TESTING A S S E S S M E N T PROBLEM A7.3 Coin Pro d u c t i o n A7.3 page 1 of 4 A very accurate machine is needed for the production of coins. The variability in weight and dimension has to be very small because coins are used in all kinds of machines. Although the production is quite accurate, 15% of all coins still don t meet the standards for weight and dimension. After pro d u c t i o n, every coin is tested for both weight and dimension. The test recognizes 90% of the good coins, and fails to recognize 20% of the bad coins. This information is displayed in the tree diagram (Figure 1). Figure 1. Tree diagram for coin production. TG = True Good FB = False Bad FG = False Good TB = True Bad Ten thousand coins are produced daily. 1. In a daily production, how many coins will test as good? 2. What percentage of the good coins will be marked as bad by the test? 6 4 9

A S S E S S M E N T Unit 7: IMPERFECT TESTING Mathematics: Modeling Our World A7.3 page 2 of 4 A t ree is a helpful tool when you are doing your calculations. The outcomes of these calculations, even without actually doing them, can be made visible in an area model. For this area model, use a square that is subdivided into 100 small squares. Every small square re p resents 1% or, in the case of the coins, can be considered as 100 coins produced. The information about the p roduction of good and bad coins is written across the top line of the square (see the left-hand square in Figure 2). The information about the test is written on both sides (left and right) of the square. You can see this in the right-hand s q u a re in Figure 2. Figure 2. Two area models. Placing product and test information in this way divides the square into four regions. Each one of the regions represents one of the four possible test results. 3. Why are the numbers 0.9 and 0.1 placed on the left side of the square? Is it possible to exchange the numbers on the right side with the ones on the left side? Label the test information numbers with their meanings (as was done for you with product information). 4. Put the letters that were used in the tree (TG, TB, FG, and FB) in the correct regions on the right-hand square in Figure 2. 5. Shade the areas of the right-hand square that represents the answer to Item 1. 6 5 0

Mathematics: Modeling Our World Unit 7: IMPERFECT TESTING A S S E S S M E N T 6. How many coins will test as bad each day even though they were produced correctly? Read the answer from the area model, and check your answer by calculation. A7.3 page 3 of 4 Now consider a new situation. A better (and more expensive) machine is used for the production of coins: 90% of all coins a re good. However, the test used to check the coins pro d u c e d is cheaper than the former one. By this test, only 80% of all good coins are seen as good. 7. Use the empty square in Figure 3 to design the area model for this new situation. Coins that test as bad will be melted, so they can be used again for the production of new coins. The cost of the better machine (90% good coins) is equal to the savings from the cheaper test. The number of bad coins that pass the test should be as small as possible. 8. Which one of the two possibilities would you advise for production: the one that is represented in Figure 2 or the one that is represented in Figure 3? Use the information given above to defend your choice. Figure 3. Empty square for new area model. The quality of the production of coins depends on two important criteria: (A)The number of bad coins in the total number of coins that pass the test as good may not exceed 3%. This is a strict government standard. (B)The percentage of coins that are seen as bad by the test must not exceed 15%. Otherwise, too many coins have to be melted each day. 9. Find out if the productions represented in Figures 2 and 3 meet these two criteria. Explain your findings. 6 5 1

A S S E S S M E N T Unit 7: IMPERFECT TESTING Mathematics: Modeling Our World A7.3 page 4 of 4 The quantity of production can be influenced in diff e rent ways: by buying b e t t e r, but more expensive, machines or by improving test re l i a b i l i t y. A combination is also possible. In the area model, these changes are visible as a shift of the lines that divide the square into four regions (see Figure 4). A s h i f t of one of the three lines (numbered 1, 2, and 3) says something about a change in the production or the test. 10. What is changing if line 1 shifts to the left? What are the consequences for the two criteria (A) and (B) if line 1 is shifted to the left? What about a shift to the right? 11. Discuss the consequences if lines 2 and 3 are shifted up or down. Figure 4. Shifting lines. 12. Advise the staff of the coin factory about changes in their production and/or testing. Use area models and trees to illustrate your ideas. Keep in mind that business people don t like to spend money if they don t see any profits! 6 5 2

Mathematics: Modeling Our World Unit 7: IMPERFECT TESTING A S S E S S M E N T PROBLEM A7.4 Test Tu b e s A7.4 page 1 of 2 If an animal in a herd has a contagious disease, it can be isolated to protect the other animals. However, some diseases can be transmitted before visible symptoms appear. Scientists have developed methods for early diagnosis by checking blood samples, but these tests are still unre l i a b l e. For a certain disease, a test is available that can: correctly identify infected animals in 80% of these cases, and correctly identify healthy animals in 90% of those cases. 1. Describe the situation with a tree diagram. Blood samples have been taken from all 225 animals of a herd. In a lab, the test tubes are placed in a rack. After the addition of a special liquid two diff e rent colors are visible. In the picture of the rack in Figure 1, you see the 225 tubes. The white ones are identified as healthy, the black ones are identified as infected. 2. Estimate the number of animals that are really infected. Figure 1. Test tubes. 6 5 3

A S S E S S M E N T Unit 7: IMPERFECT TESTING Mathematics: Modeling Our World A7.4 page 2 of 2 The blood test mentioned previously can only indicate whether an animal has the disease or not. Infected animals are isolated from the others but are not t reated for the disease. There is another, more effective approach for fighting the disease. Some infected animals produce a large amount of antibodies and can there f o re re c o v e r. So the new goal for the re s e a rchers is enhancement of the production of antibodies with the help of medicines. To study the effect of the medicine, the researcher did an experiment with two groups of animals, all showing early symptoms of the disease. Group A was treated with the medicine, and Group B was not treated with the medicine. All animals were tested for the amount of antibodies that were produced with (Group A) or without the medicine (Group B). Figure 2 shows the result of the experiment. Black: There is a sufficient amount of antibodies in the blood. White: The amount of antibodies is insufficient. Figure 2. Comparing two groups. 3. What do you think about the effect of using the medicine for the production of antibodies? Explain your answer. 6 5 4

Mathematics: Modeling Our World Unit 7: IMPERFECT TESTING A S S E S S M E N T PROBLEM A7.5 Test Results 1. A test for a performance-enhancing drug correctly detects 90% of the drug users and correctly reports the absence of drugs in 99% of the cases. 10,000 people are tested, among which 1000 are drug users. a) How many of the drug users do you expect will go undetected? A7.5 page 1 of 2 b) How many of these people tested do you expect will test positive for drug use although they are not actually users? c) What percentage of those who test positive can you expect to be actually drug-free? 2. Suppose that the probability is.9 that a test will correctly detect a true positive or a true negative. Also, suppose that 5% of the people tested have a certain trait for which you are testing. Now you can predict the percentage of tests that will have a positive result. Do you expect that the test result will be that 5% are positive? If your answer is yes, give an explanation. If your answer is no, what percentage of positives do you expect will result from the test? 3. When you know the quality of a test, the test results give you information about the population. Suppose that the percentage of people with a certain defect is unknown. When people with that defect are tested, the test gives a positive result in 90% of the cases. For people who do not possess that defect, the test reports the absence of that defect in 90% of the cases. A large number of tests result in 20% positive tests (reporting that the individuals tested have that defect). What percentage of the people tested really do have that defect? 6 5 5

A S S E S S M E N T Unit 7: IMPERFECT TESTING Mathematics: Modeling Our World A7.5 page 2 of 2 4. Assume that a performance-enhancing-drug test is fairly reliable: the probability that the test correctly detects the presence of drugs is.95, and the probability of correctly reporting the absence of drugs is.99. a) If 10,000 athletes are tested and 1% of them are using the illegal drugs, how many athletes do you expect will test positive? b) What percentage of those positives do you expect will be real users? c) How many athletes do you expect tested positive but actually did not use drugs? 6 5 6