But How Do You Know It s Working?

Transcription

1 But How Do You Know It s Working? Outcome Measurement and Evaluation MARY HANIFIN SENIOR MANAGING DIRECTOR, DEVELOPMENT MASSACHUSETTS GENERAL HOSPITAL LABRINI NELLIGAN EXECUTIVE DIRECTOR LUNDER- DINEEN HEALTH EDUCATION ALLIANCE OF MAINE

2 Agenda 1. Introduction/Learning Outcomes 2. Why Do Outcomes Matter? 3. Definitions 4. Logic Model in Theory 5. Logic Model in Practice 6. Discussion

3 Learning Outcomes At the end of this presentation, you will be able to: Apply a logic model template to a Development case scenario Examine best practices for outcome measurement and evaluation Identify action steps to advance a skill set or partnership in outcome measurement and evaluation on your team

4

5 What famous author was born in Maine? A.Henry David Thoreau B.Stephen King C. Edith Wharton D.James Irving A B C D 0

6 Smart Poll Question 1 I know what the core components of a logic model are. A. Strongly Agree B. Agree C. Disagree D. Strongly Disagree

7 Smart Poll Question 2 I am confident in my ability to work with a program and/or research partner(s) on outcome measurements and evaluation. A. Strongly Agree B. Agree C. Disagree D. Strongly Disagree

8

9 Describe the study methods. Include information about study groups, methods, measures, sample collection, time points of collection, statistical analysis, etc. This is the most important section and should be the primary focus of the concept paper. Describe how results will benefit clinical practice (how will a practicing clinician use this information) and provide a projection of the number of years before findings may result in significant changes in practice after the study is complete. The Foundation strongly believes that, without the intentional application of a health equity lens, proposed activities may not achieve intended results or may unintentionally end up maintaining or worsening disparities. Applicants invited to submit a full proposal should be prepared to explicitly describe the impact that their activities will have in the promotion of health equity. We are interested in hearing about outcomes. Did your project make a difference to your community or your organization? If there was an evaluation done, either in-house or independently, we would like to see the results Interim reports should include an on-going evaluation of the project and a description of any changes from the information shown in the original grant application. This description must cover changes in actual or projected budgets, revenue, or funding sources.

10 Who s on the Team? Development Officer Program Manager Physician Subject Matter Expert Researcher

11 Show Your Work : the Logic Model

12 Logic Models: What are they? [A logic model is] a systematic and visual way to present how you believe your program will work. Kellogg Foundation Logic models allow you to draw connections among three key elements: 1) the resources needed to run your organization s programs; 2) the organization s activities; and 3) the long-term impact that your organization is trying to achieve. Network for Good A logic model shows the route traveled (or steps taken) to reach a certain destination. Center for Community Health and Development, Kansas University

13 Logic Models: Why should we care? Some External Reasons: Fundraising Marketing/Communications Accountability Transparency Some Internal Reasons: Stakeholder Buy-In Program Management Staff Alignment Outcomes Data

14 Logic Model Your Planned Work Your Intended Results Resources/Inputs Activities Outputs Outcomes Impact

15 A Simple Analogy: The Case for High Top Sneakers Credit: Understanding Logic Models by way of a simple analogy Accelerate Now with Michael Brand

16 Logic Model in Practice

17 [ ] Health Education is Powerful Medicine Lunder-Dineen offers free, easily accessible and evidence-based education to Maine health care professionals and the communities they serve through an innovative partnership with Massachusetts General Hospital and an ongoing collaboration with Maine s health care community.

18 "When in distress every man becomes our neighbor ~1811 Circular letter

19 Building Blocks Form and Cultivate Relationships Identify and support future leaders Continually Raise Awareness Build a Solid Framework Establish Academic Partnerships Leverage Innovative Learning Methodologies and Technologies Identify and Align with Stakeholders Identify Areas of Educational Priority

20 Three Strategic Projects

21 Concept Providing healthcare professionals with knowledge, skills and attitudes to engage in meaningful conversations about alcohol use and supporting the primary care team in establishing best practices, will help to advance compassionate and outcome-based patient care.

22 Needs Assessment and Evaluation Partner

23 Evaluation Goal Identify strengths and opportunities for improvement in the pilot program

24

25 - Sufficient project funding - Team of subject matter experts - Engaged practice site to test project - Needs Assessment and Data to inform program design - Technology resources to support program - Assemble team of site champions for project -Develop program framework - Provide Education and Training to the interprofessional team - Recommend best practices to operationalize the program - Develop a network of support for the providers and the practice sites Produce Unhealthy Alcohol Use Education Program Empower health care providers (learners) with the knowledge, skills, and attitudes to confidently engage and address unhealthy alcohol use with their patients Early intervention before patients are at greater risk may reduce likelihood of developing substance use disorders and associated chronic illnesses (diabetes, cancer) leading to better health outcomes

26 Program Development Framework Convene Stakeholders Needs assessment Analyze findings Design and develop program Implementation Evaluation Integration of findings Expansion

27 - Sufficient project funding - Team of subject matter experts - Engaged practice site to test project - Needs Assessment and Data to support project - Technology resources to support program - Assemble team of site champions for project - Provide Education and Training to the interprofessional team - Provide Practice Facilitation bestpractices - Provide Expert Consultation resources -12-person Advisory Team assembled and convened monthly - 3 Education Modules - 40 health care professionals trained at the site - 6,500 patients reached - Empower health care providers (learners) to confidently engage and address unhealthy alcohol use with their patients - Increase in early intervention tactics in primary care provision - Refer more people to specialized treatment

28 Major Components of an Evaluation Framework A. Collecting Data B. Summarizing Conclusions C. Create a Lessons Learned Report

29 Numbers versus Narrative It s not Numbers versus Narrative, it s both! Sources:

30 Picking the Right Evaluation Partner Survey Development Skills Development of Action Plan Collecting Data (Mixed-Methods) Expertise in the subject matter Survey Software Portfolio Survey Analytics Customer Service Business Agreements

31 Lessons Learned Develop a Concept Start Evaluation Process at the Beginning Customize a Model that works for You Seek the help of a researcher and/or develop skills on the team Develop a strong relationship between Development and Program/Physician Maintain Reputation Manage your donor s expectations

32 Smart Poll Question 1 I know what the core components of a logic model are. A. Strongly Agree B. Agree C. Disagree D. Strongly Disagree

33 Smart Poll Question 2 I am confident in my ability to work with a program and/or research partner(s) on outcome measurements and evaluation. A. Strongly Agree B. Agree C. Disagree D. Strongly Disagree

34 Discussion

35 Bibliography Malloy, Laurel. Finally Outcome Measurement Strategies Anyone Can Understand. Second Edition, Festen, Marcia, and Marianne Philbin. Level Best: How Small and Grassroots Nonprofits Can Tackle Evaluation and Talk Results. John Wiley and Sons, Inc., Chardon Press, Penna, Robert M. The Nonprofit Outcomes Toolbox: A Complete Guide to Program Effectiveness, Performance Measurement, and Results. John Wiley and Sons, Inc., Using Logic Models to Bring Together Planning, Evaluation, and Action: Logic Model Development Guide. W. M. Kellogg Foundation, 2004

36 Acknowledgements Kate Dixon, Director of Development, Corporate and Foundation Relations, MGH Kara Donnelly, Director of Development, Corporate and Foundation Relations, MGH Samantha Nock, Program Coordinator, Lunder-Dineen Health Education Alliance of Maine, MGH Carole MacKenzie, M.Ed., BSN, RN-BC, Professional Development Specialist, Lunder-Dineen Health Education Alliance of Maine, MGH Thank you!