Program Evaluation: Methods and Case Studies Emil J. Posavac Eighth Edition

Size: px

Start display at page:

Download "Program Evaluation: Methods and Case Studies Emil J. Posavac Eighth Edition"

Charlotte Wilkinson
5 years ago
Views:

1 Program Evaluation: Methods and Case Studies Emil J. Posavac Eighth Edition

2 Pearson Education Limited Edinburgh Gate Harlow Essex CM20 2JE England and Associated Companies throughout the world Visit us on the World Wide Web at: Pearson Education Limited 2014 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without either the prior written permission of the publisher or a licence permitting restricted copying in the United Kingdom issued by the Copyright Licensing Agency Ltd, Saffron House, 6 10 Kirby Street, London EC1N 8TS. All trademarks used herein are the property of their respective owners. The use of any trademark in this text does not vest in the author or publisher any trademark ownership rights in such trademarks, nor does the use of such trademarks imply any affiliation with or endorsement of this book by such owners. ISBN 10: ISBN 10: ISBN 13: ISBN 13: British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library Printed in the United States of America

3 criteria for assessing the work of mental health workers who screen people seeking care. When using multiple variables, it is important to use different kinds of variables expected to be affected by different sources of error. Variables measured in very similar ways from one source may be affected by the same biases. When this happens, the extra effort is of little value. By using several variables, assessed in different ways from different sources of data, evaluators can draw conclusions from the convergence of these variables and thus add considerable strength to an evaluation. Case Study, Using Multiple Measures in an Evaluation of a Summer Community Program for Youth, illustrates how the convergence of multiple measures provided strong support for a summer program for teenagers. Use Nonreactive Measures Nonreactive measurement refers to procedures of gathering information that do not prompt the respondent to give particular answers. For example, whenever it seems obvious that certain answers are desired by interviewers, some respondents tend to give those answers. The previous section hinted at another problem of reactive measures: Merely focusing attention on a variable can lead people to change their behavior. Even surveys can be reactive because they may prompt respondents to think about issues they may have never concerned themselves with before. Clearly when an evaluation leads to changes in the behavior of staff or clients, the evaluation does not accurately reflect the program in operation. Unless the evaluation depends solely on existing records, totally nonreactive measurement is an ideal seldom achieved, although care in designing evaluations can reduce reactivity (Webb, Campbell, Schwartz, Sechrest, and Grove, 1981). Use Only Variables Relevant to the Evaluation Because program evaluation is an applied discipline, the variables selected must be relevant to the specific informational needs of facility management, community representatives, those responsible for budgets, and other stakeholders. In planning an evaluation, evaluators seek to learn what decisions are pressing but not understood. Discussions with staff and managers often reveal the important variables. If no decisions can be affected by a variable, or if no program standards mention the variable, then it probably is not essential to the evaluation. Evaluators take care not to measure variables that some stakeholders might think would be interesting but which would not affect the operation of the program or agency; such variables raise the cost of an evaluation but do not contribute to its value. Sometimes evaluators expect that a variable would be seen as important once summarized for agency leaders; at those times experienced evaluators act in the role of educator. Furthermore, evaluators seek to be sure that they are sensitive to good and bad side effects that cannot be anticipated. 97

4 CASE STUDY 1 Using Multiple Measures in an Evaluation of a Summer Community Program for Youth The value of making observations from a variety of perspectives was illustrated by an evaluation of a community policing project (Thurman, Giacomazzi, and Bogen, 1993). Note that participant observations, focus groups, telephone surveys, and written questionnaires were used and program cost was presented. Note also that the interests of four stakeholder groups participants, parents or guardians, staff, and taxpayers were examined. Police officers worked with a program designed to help 325 high-risk children develop a sense of responsibility, envision occupational roles, recognize the relationship between education and finding a good job, and hold positive images of police officers. The children, whose median age was 13, were nominated by school counselors from the most economically disadvantaged neighborhoods. The week-long program included (a) work periods spent removing trash, weeds, and graffiti; (b) lunches in parks; (c) tours of businesses and museums; (d) a $40 bank account for program completion; and (e) contact with role models, including police officers. Five focus groups were used to assess immediate reactions of the children. Each session was held near the end of the week and lasted one hour. The children volunteered comments that seemed to match the goals of the program: They said that they learned responsibility, were encouraged to complete their education, and for the most part reported positive feelings toward the police officers. The parents or guardians were randomly selected for structured telephone interviews after the program ended; 94 percent agreed to be interviewed. All said that their children had talked about positive aspects of the program and wanted to participate again in the following summer. Nearly all respondents (90 percent) said there were no negative aspects of the program and about half said that their child did gain a sense of responsibility. At the end of the summer all program staff members received surveys. Respondents were aware of organizational problems; however, nearly all had a very positive perception of the program, all recommended that the program be repeated the next summer, and all said that they wanted to be involved if it were offered again. The cost of the program was $46,311, or $ per child. There was no way to compare this cost to the long-term benefits; however, if such efforts motivate only a small number of high-risk children to be more receptive to future encouragement to follow healthy lifestyles and to avoid gangs and drugs, considerable suffering would be avoided and society would save significant future law enforcement, legal, and rehabilitation funds. Use Valid Measures The instruments must validly measure the behaviors that the program is designed to change. These behaviors may be the ultimate criterion behaviors (such as a healthy life, developed skills, employment) or more immediate behaviors believed to be useful in achieving the planned long-term outcomes (such as more consistent adherence to prescribed medical treatment, higher 98

5 quality work, or improved leadership practices). In order to increase the credibility of the evaluation, evaluators make sure that staff members have approved the measures. But because program staff members are often inexperienced in social science methodology and cannot judge how feasible it is to measure a particular variable, evaluators do not depend on them to suggest all the measures and variables. In general, the more a measurement tool focuses on objective behavior rather than on ill-defined or vague terms, the more likely it is to be valid. Objective behaviors (such as late to work less often than once a week or speaks in groups ) can be more validly measured than traits (such as punctual and assertive ), regardless of the specific measurement approach chosen (Mager, 1997). Even when the variables seem quite objective, information taken from official records may not be as valid as it first appears. Changing definitions and differing levels of care in preparing records can introduce changes in the meaning of variables that could influence an evaluation. At least twice Chicago s crime records appeared to reveal an increase in crime when better recordkeeping was begun by the police. After a well-respected person was installed as chief of police, thefts under $50 increased dramatically. Instead of ushering in a crime wave, the new chief simply insisted on better reporting procedures, and may have reclassified some crimes (Campbell, 1969). Since homicides are usually recorded accurately and cannot be reclassified into a different category, homicide rates were used to determine that the crime rate did not increase when the new chief took over. In 1983 a similar increase was observed when recording standards were again strengthened ( Good News, 1983). After the renorming of the SAT, average SAT scores of college freshmen showed an increase between 1994 and 1995, a result solely due to a change in scaling by the Educational Testing Service ( Notebook, 1994). The national jobless rate increased in 1994 because the Bureau of Labor Statistics changed the survey used to assess joblessness to better reflect the participation of women in the workforce ( Survey s Overhaul, 1993). In 2000 the U.S. Census revised the categories used by respondents to indicate their race (Lee, 2001). This change means that information on race from the 2000 Census cannot be directly compared with previous census data. Using records is efficient and often less expensive than making observations specifically for an evaluation, but whenever one uses data collected for a purpose other than the evaluation, a critical examination of the applicability of the data is always in order. Use Reliable Measures The concept of reliability. Reliability refers the consistency of the information that a measure produces for the same person in the same situation (Murphy and Davidshofer, 2005). There are many ways to think about reliability. If different observers describing the same thing report similar levels of 99

6 a variable, we say that the observation procedure is reliable. If different levels are reported, we say that the procedure is unreliable. Reliability is calculated using the correlation coefficient, r. If two observers agreed completely after making a number of observations, a correlation of the two sets of data would be 1.00, perfect reliability. To the extent that they do not agree with each other, the correlation would be smaller; 0.0 would indicate that knowing one observer s reports would tell us nothing about the other observer s reports. If an outcome measure is a 50-item test of knowledge gained from, say, a job training program, there is another way to estimate reliability. The scores that a group of trainees earned on half of the items (say, the odd-numbered items) can be correlated with the scores for the other half of the items; such a correlation is called a split-half reliability. Note that the focus of interest is still consistency just as in the first example concerning interobserver reliability. For split-half reliability the issue is inter-item consistency, that is, the different items seem to be measuring the same thing. Statistical analysis programs make such calculations simple to do (Kirkpatrick and Finney, 2008). Reliability is higher when the measurement procedure is minimally affected by the passing moods of the observer or the individuals observed. Like validity, reliability is likely to be higher when observations are based on objective behaviors rather than on inferences. Reliability is also higher when the measurement instrument contains more items. You may well have a sense of the higher reliability produced by using many questions if you think of a classroom test. A single item forms a larger portion of a short test than it would of a long test. If you had happened to have forgotten the topic covered by a certain question, your score would be affected much more in the context of a short test than in the context of a long test. Because a single question can have major effect, short tests are less reliable than long tests. The more reliable a measure is, the more likely it is to detect an effect of a program (Lipsey, 1990). Evaluators trained in disciplines in which the individual assessment is important are familiar with the reliability of classroom, achievement, or job selection tests. It is crucial that evaluators recognize that using tests to evaluate individuals is markedly different from using tests to evaluate programs. Since evaluators focus attention on groups, not individuals, surveys and observational techniques that would not be useful for individual assessment are quite acceptable to estimate the mean score of a group. By using the standard deviation of a measure and its reliability, it is possible to calculate a range of values likely to include a person s true score. For example, assume a person scored 105 on an IQ test whose standard deviation is 15 and reliability is The true scores of 68 percent of the people getting 105 will be between 105 ; , or and The value added and subtracted to the person s score is called the standard error of measurement and applies when measuring an individual s score (Murphy and Davidshofer, 2005). This range is called a confidence interval. 100

7 By contrast, when estimating a group s mean score, the standard error of the mean should be used to calculate confidence intervals instead of the standard error of measurement. Suppose a group of 100 students are selected at random from a school. Suppose further that they have an average of 105 on the same IQ test mentioned above. For a group, we work with the standard error of the mean which, in this case, is , or We can conclude that 68 percent of such groups have true mean scores between 105 ; , or and Compare this confidence interval with the one found for an individual in the paragraph above. We can be more certain of the likely value of a group mean than the score of any one of the individuals in the group. Even a measure with a reliability of only 0.40 would yield a sufficiently precise estimate for program evaluation purposes because evaluators are concerned about groups, not individuals. Even short scales are often sufficiently reliable for evaluating the effects of college experiences when samples are large (Walvoord, 2004). Different forms of reliability. Some reliability indexes are less relevant to evaluation than to individual assessment. A crucial distinction relates to the difference between assessing a personality trait that is thought to be a stable characteristic of an individual and measuring a behavior that can change, depending on the intentions of people or the situations in which they act. For personality assessment, a high test retest reliability shows that the measure reliably produces consistent estimates of a trait. The items on such measures have been chosen because they produce stable results; however, stability is not the focus of evaluators. Programs are designed to change people or communities; program success cannot be evaluated with measures that were selected to be stable. Evaluators seek measures that are sensitive to change (Lipsey and Cordray, 2000). A form of reliability that evaluators seek in scales is homogeneity, that is, the degree to which a scale measures one thing rather than a mixture of behaviors. If a reading program is to be evaluated, the instrument should measure reading improvement, not general knowledge or quantitative skills. Cronbach s alpha is available in many statistical programs and is often reported when several items are placed together to form a scale. In essence, this index is a generalization of the traditional split-half reliability coefficient without the need to divide the scale into arbitrary halves (Pedhazur and Schmelkin, 1991). When observer ratings are used, evaluators also seek high interobserver reliability. An instrument that yields similar values when used by different people has more credibility than one that yields different values. Obviously, if the ratings are done in different settings at home versus at school a low interobserver reliability index may be observed due to real differences between the settings themselves. Interobserver reliability can be increased through the development of a detailed instruction manual and careful training and periodic monitoring of observers. 101

Research in Education. Tenth Edition

Research in Education. Tenth Edition Research in Education John W. Best James V. Kahn Tenth Edition Pearson Education Limited Edinburgh Gate Harlow Essex CM20 2JE England and Associated Companies throughout the world Visit us on the World