But How Do You Know It s Working? Outcome Measurement and Evaluation MARY HANIFIN SENIOR MANAGING DIRECTOR, DEVELOPMENT MASSACHUSETTS GENERAL HOSPITAL LABRINI NELLIGAN EXECUTIVE DIRECTOR LUNDER- DINEEN HEALTH EDUCATION ALLIANCE OF MAINE
Agenda 1. Introduction/Learning Outcomes 2. Why Do Outcomes Matter? 3. Definitions 4. Logic Model in Theory 5. Logic Model in Practice 6. Discussion
Learning Outcomes At the end of this presentation, you will be able to: Apply a logic model template to a Development case scenario Examine best practices for outcome measurement and evaluation Identify action steps to advance a skill set or partnership in outcome measurement and evaluation on your team
www.mghdevelopment.participoll.com
What famous author was born in Maine? A.Henry David Thoreau B.Stephen King C. Edith Wharton D.James Irving A B C D 0
Smart Poll Question 1 I know what the core components of a logic model are. A. Strongly Agree B. Agree C. Disagree D. Strongly Disagree
Smart Poll Question 2 I am confident in my ability to work with a program and/or research partner(s) on outcome measurements and evaluation. A. Strongly Agree B. Agree C. Disagree D. Strongly Disagree
Describe the study methods. Include information about study groups, methods, measures, sample collection, time points of collection, statistical analysis, etc. This is the most important section and should be the primary focus of the concept paper. Describe how results will benefit clinical practice (how will a practicing clinician use this information) and provide a projection of the number of years before findings may result in significant changes in practice after the study is complete. The Foundation strongly believes that, without the intentional application of a health equity lens, proposed activities may not achieve intended results or may unintentionally end up maintaining or worsening disparities. Applicants invited to submit a full proposal should be prepared to explicitly describe the impact that their activities will have in the promotion of health equity. We are interested in hearing about outcomes. Did your project make a difference to your community or your organization? If there was an evaluation done, either in-house or independently, we would like to see the results Interim reports should include an on-going evaluation of the project and a description of any changes from the information shown in the original grant application. This description must cover changes in actual or projected budgets, revenue, or funding sources.
Who s on the Team? Development Officer Program Manager Physician Subject Matter Expert Researcher
Show Your Work : the Logic Model
Logic Models: What are they? [A logic model is] a systematic and visual way to present how you believe your program will work. Kellogg Foundation Logic models allow you to draw connections among three key elements: 1) the resources needed to run your organization s programs; 2) the organization s activities; and 3) the long-term impact that your organization is trying to achieve. Network for Good A logic model shows the route traveled (or steps taken) to reach a certain destination. Center for Community Health and Development, Kansas University
Logic Models: Why should we care? Some External Reasons: Fundraising Marketing/Communications Accountability Transparency Some Internal Reasons: Stakeholder Buy-In Program Management Staff Alignment Outcomes Data
Logic Model Your Planned Work Your Intended Results Resources/Inputs Activities Outputs Outcomes Impact
A Simple Analogy: The Case for High Top Sneakers Credit: Understanding Logic Models by way of a simple analogy Accelerate Now with Michael Brand
Logic Model in Practice
[ ] Health Education is Powerful Medicine Lunder-Dineen offers free, easily accessible and evidence-based education to Maine health care professionals and the communities they serve through an innovative partnership with Massachusetts General Hospital and an ongoing collaboration with Maine s health care community.
"When in distress every man becomes our neighbor ~1811 Circular letter
Building Blocks Form and Cultivate Relationships Identify and support future leaders Continually Raise Awareness Build a Solid Framework Establish Academic Partnerships Leverage Innovative Learning Methodologies and Technologies Identify and Align with Stakeholders Identify Areas of Educational Priority
Three Strategic Projects
Concept Providing healthcare professionals with knowledge, skills and attitudes to engage in meaningful conversations about alcohol use and supporting the primary care team in establishing best practices, will help to advance compassionate and outcome-based patient care.
Needs Assessment and Evaluation Partner
Evaluation Goal Identify strengths and opportunities for improvement in the pilot program
- Sufficient project funding - Team of subject matter experts - Engaged practice site to test project - Needs Assessment and Data to inform program design - Technology resources to support program - Assemble team of site champions for project -Develop program framework - Provide Education and Training to the interprofessional team - Recommend best practices to operationalize the program - Develop a network of support for the providers and the practice sites Produce Unhealthy Alcohol Use Education Program Empower health care providers (learners) with the knowledge, skills, and attitudes to confidently engage and address unhealthy alcohol use with their patients Early intervention before patients are at greater risk may reduce likelihood of developing substance use disorders and associated chronic illnesses (diabetes, cancer) leading to better health outcomes
Program Development Framework Convene Stakeholders Needs assessment Analyze findings Design and develop program Implementation Evaluation Integration of findings Expansion
- Sufficient project funding - Team of subject matter experts - Engaged practice site to test project - Needs Assessment and Data to support project - Technology resources to support program - Assemble team of site champions for project - Provide Education and Training to the interprofessional team - Provide Practice Facilitation bestpractices - Provide Expert Consultation resources -12-person Advisory Team assembled and convened monthly - 3 Education Modules - 40 health care professionals trained at the site - 6,500 patients reached - Empower health care providers (learners) to confidently engage and address unhealthy alcohol use with their patients - Increase in early intervention tactics in primary care provision - Refer more people to specialized treatment
Major Components of an Evaluation Framework A. Collecting Data B. Summarizing Conclusions C. Create a Lessons Learned Report
Numbers versus Narrative It s not Numbers versus Narrative, it s both! Sources: https://gallery.mailchimp.com/627810fb1d8daa951beafa7e5/files/479c953f-b8fc-4f5e-9096-e6463dc35cdd/framing_stories_for_change.pdf https://gallery.mailchimp.com/627810fb1d8daa951beafa7e5/files/3bd9e714-e400-49f0-8307-13faabe3d1fc/the_storytelling_power_of_numbers.pdf
Picking the Right Evaluation Partner Survey Development Skills Development of Action Plan Collecting Data (Mixed-Methods) Expertise in the subject matter Survey Software Portfolio Survey Analytics Customer Service Business Agreements
Lessons Learned Develop a Concept Start Evaluation Process at the Beginning Customize a Model that works for You Seek the help of a researcher and/or develop skills on the team Develop a strong relationship between Development and Program/Physician Maintain Reputation Manage your donor s expectations
Smart Poll Question 1 I know what the core components of a logic model are. A. Strongly Agree B. Agree C. Disagree D. Strongly Disagree
Smart Poll Question 2 I am confident in my ability to work with a program and/or research partner(s) on outcome measurements and evaluation. A. Strongly Agree B. Agree C. Disagree D. Strongly Disagree
Discussion
Bibliography Malloy, Laurel. Finally Outcome Measurement Strategies Anyone Can Understand. Second Edition, 2010. Festen, Marcia, and Marianne Philbin. Level Best: How Small and Grassroots Nonprofits Can Tackle Evaluation and Talk Results. John Wiley and Sons, Inc., Chardon Press, 2007. Penna, Robert M. The Nonprofit Outcomes Toolbox: A Complete Guide to Program Effectiveness, Performance Measurement, and Results. John Wiley and Sons, Inc., 2011. Using Logic Models to Bring Together Planning, Evaluation, and Action: Logic Model Development Guide. W. M. Kellogg Foundation, 2004
Acknowledgements Kate Dixon, Director of Development, Corporate and Foundation Relations, MGH Kara Donnelly, Director of Development, Corporate and Foundation Relations, MGH Samantha Nock, Program Coordinator, Lunder-Dineen Health Education Alliance of Maine, MGH Carole MacKenzie, M.Ed., BSN, RN-BC, Professional Development Specialist, Lunder-Dineen Health Education Alliance of Maine, MGH Thank you!