Research Strand: Program Evaluation. Stephen Kroeger, Christy Borders, Kim Webster

Transcription

1 Research Strand: Program Evaluation Stephen Kroeger, Christy Borders, Kim Webster

2 Logicizing University of Cincinnati The Birth of a Logic Model to Evaluate one 325T Program

3 Answers Needed The Government Performance and Results Act (GPRA) essentially asks three questions: What is our program trying to achieve? How will our program s effectiveness be determined? How is our program actually doing? The Challenge Clear and logically consistent methods have not been readily available to help program coordinators make implicit understandings explicit (McLaughlin & Jordan, 1999).

4 Measuring Change Providers are often being required to document specific interventions and expected outcomes and to demonstrate their efforts to increase program effectiveness. As a result, a great number of providers need to identify services or policies that are designed to lead to positive changes in the services provided to children and families they serve. Yampolskaya, Nesman, Hernandez, & Koch, 2004

5 What is a Logic Model process? The Logic Model process is a tool used by program evaluators and coordinators to develop and tell the performance story for their program. The Logic Model describes the reasonable linkages among program resources, activities, outputs, stakeholders reached, and short-intermediate-longer term outcomes. Once this model of expected performance is produced, critical measurement areas can be identified. (McLaughlin & Jordan, 1999)

6 Benefits of a Logic Model process Builds a common understanding of the program, identifies assumptions, and supports team communication Identifies projects that are critical to goal attainment, redundant, or have inconsistent or implausible linkages among program elements Communicates the place of a program in the organization or problem hierarchy Provides a balanced set of key performance measurements and evaluation issues to improve data collection and usefulness Meets requirements of GPRA (McLaughlin & Jordan, 1999).

7 Logic Model Models Gugiu & Rodriguez-Campos, 2007 Theory Approach logic model asks four questions: What issues or problems does the program seek to address? What are the specific needs of the target audience? What are the short- and long-term goals of the program? What barriers or supports may impact the success of the program? W. K. Kellogg foundation logic models United Way of America logic model consists of resources needed to achieve goals; activities implemented to affect change; outputs are services provided, materials distributed, people served; and outcomes are derived benefits as a result of program activities. Contextual factors may be incorporated when it s important to note constraints on the program. Antecedent Target Measurement (ATM) logic models identify antecedent conditions on a problem s underlying rationale (why); identify antecedent conditions targeted by program activities; determine whether outcomes are reasonable to include in the logic model in light of the evaluation timeframe. Open Systems logic models focus on establishing collaborative partnerships to achieve strategic objectives and to measure impacts at various levels. Levels include changes that are temporary; sustaining the prior changes and stakeholder buy-in; making broader changes in organizational practices prevent/reduce problems.

8 Learning Curve During Construction Activities are not Goals We had over 10 activities we thought were goals These activities were categorized into 4 main goals Outcomes Link Back to Activities Whether Short term, Intermediate or Long term, the outcomes need to provide data used to evaluate the activity Outcomes Must be Measureable If we can t quantify or qualify the outcome we won t know if the activity was useful or effective The act of establishing outcome measures required that we modify the activities to produce measureable outcomes What activity would increase retention of new teachers?

9 Logic Model Changed Our Thinking

10 Logic Model Changed Our Thinking

11 Logic Model Changed Our Thinking What product will come out of this activity? Changes or benefits caused by outputs

12 Logic Model Changed Our Thinking Short-term = changes or benefits caused by outputs. Intermediate = results as an application of short-term Long-term = are these results correct?

13 Logic Model Changed Our Thinking Data: Survey Collection: Yearly Feedback: Failure to meet outcome would result in the examination of how we are communicating instruction on EBP What data? When/how to collect? How will this feedback to activities?

14 Daily Use of the Logic Model How can we use the Logic Model to inform day-to-day decisions? Funneling the goals into daily tasks in a manageable format Goals Activities Daily Tasks

15 Feedback Loops Data provides continuous feedback. Resources, activities, outputs, customers, or outcomes may adjust based on the how the data are supporting the project s goals.

16 Sample of Daily Graphic

17 Questions How does a logic model help me? What resources do I need in order to begin using a logic model? Can you give an example of how the LM helped you in your project design and evaluation?

18 References Gugiu, P. C., & Rodriguez-Campos, L. (2007). Semistructured interview protocol for constructing logic models. Evaluation and Program Planning, 30, McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: A tool for telling your program's performance story. Evaluation and Program Planning, 22, Yampolskaya, S., Nesman, T. M., Hernandez, M., & Koch, D. (2004). Using concept mapping to develop a logic model and articulate a program theory: A case example. American Journal of Evaluation, 25(2),