Workshop Presented at University of Maryland Training Institutes, July 25-28,2018, Washington, D. C.

Transcription

1 How to Continuously Evaluate System of Care Expansion by Combining Universal Big Data From All Human Services and Entire School Populations: Examples From Chautauqua & Rockland Counties (New York State) & Manchester City Council (UK) Workshop Presented at University of Maryland Training Institutes, July 25-28,2018, Washington, D. C. Mansoor A. F. Kazi, PhD, President, Realist Evaluation Inc., Director, Program Evaluation Center, Fredonia SUNY; Rachel M Ludwig, LCSW, Chautauqua Tapestry Director; John Rosiak, Rosiak Associates, LLC Susan Hoerter, DO, Rockland County SOC Expansion Grant PI; Janet Sliva, Rockland County SOC Expansion Grant Manager

2 Acknowledgements Dr. Bret Apthorpe, Superintendent; Jessie Joy, Director of Curriculum, Instruction & Assessment; Tina Sandstrom, Director of Schools, Jamestown Public Schools; John Panebianco, Director of Pupil Services Dr. Gary M. Blau, Chief, Child, Adolescent and Family Branch, Center for Mental Health Services, Substance Abuse and Mental Health Services Administration Patricia Brinkman, Commissioner; Chautauqua County Department of Mental Hygiene, Mayville Rachel Ludwig, Director Chautauqua Tapestry Manchester City Council Marie McLaughlin, Director Youth Justice Services

3 Acknowledgements Ed Day, Rockland County Executive Tom Zugibe, District Attorney Michael Leitzes, Mental Health Commissioner Mary Jean Marsico, Rockland BOCES Dr. Susan Hoerter, Rockland SOC Expansion PI Janet Sliva, Rockland SOC Expansion Grant Manager Barbara Gavin, Director, Child Welfare Services Carol Korngold, Safe Youth Center Executive Director Kathy Farber, Safe Youth Center Consulting Director Kathy Tower-Bernstein, Director of Probation

4 Mansoor A. F. Kazi, President, Realist Evaluation Inc.; Affiliate Research Fellow, University at Buffalo; Director, Program Evaluation Center, Fredonia State University of New York; Adjunct Fordham University American Evaluation Association (eval.org) Co-Chair Human Services Evaluation Topical Interest Group & Social Work Topical Interest Group. Realist Evaluation Partnerships Tapestry of Chautauqua County, New York STAKES/THL MANCHESTER CITY COUNCIL, England ROCKLAND COUNTY NY SAMHSA s Gold Award for Outstanding Local Evaluation 2010 Training Institute Based on Kazi, M. A. F. (2003) Realist Evaluation in Practice, London: Sage

5 New York State System of Care SAMHSA Funding Funded by SAMHSA Substance Abuse and Mental Health Services Administration, Center for Mental Health Services: Tapestry of Chautauqua Systems of Care Grant (with Chautauqua County Department of Mental Hygiene, $9,000, ; $977,317 for evaluation, Funded by SAMHSA; Recipient of SAMHSA s Gold Award for Outstanding Local Evaluation, July 2010). Tapestry of Chautauqua Systems of Care Expansion Grant (with Chautauqua County Department of Mental Hygiene, $4,000, ) Rockland County (NY) Systems of Care Expansion Grant, with Rockland County, $4,000, ; $800,000 for evaluation, Funded by Substance Abuse and Mental Health Services Administration (SAMHSA).

6

7 Take the Gary blau challenge

8 System of Care: Focus on Values Continuous Quality Improvement Cultural & Linguistic Competence Family Driven EBP s & Clinical Excellence Transforming Children s Mental T= (V Health + B + A) in x America! (CQI) 2 Youth Guided

9 Children s Mental Health Initiative (CMHI): Systems of Care CMHI has impacted 22% of the nations counties and 33 federally recognized Indian Tribes Since 2000, SAMHSA administers an average of 61 system of care grants per year There are currently 88 CMHI funded sites CMHI sites have served over 100,000 children and youth

10 Systems Of Care Work! Reduced behavioral and emotional problems Increased behavioral and emotional skills Reduced suicidal ideation and attempts Reduced substance use problems Improved functioning in school and in the community Improved ability to build relationships

11 View all of the KSOC-TV webisodes on the SAMHSA YouTube channel or by going to

12 The Goals of a System of Care To improve access and expand the array of services and supports for children and youth with a serious emotional disturbance and their families. To promote the full potential of every child and youth by addressing their physical, emotional, intellectual, cultural and social needs in family, school & community

13 Who Comes to School? 21% of children ages 9-17 have a diagnosable mental health disorder (U.S. Surgeon General, 1999; NIMH, 20% currently) 20% of children with mental health disorders are identified and receive mental health services (U.S. Surgeon General, 2000; mentalhealth.gov 20% currently) 50% of students age 14+ who have a mental illness drop out of high school the highest dropout rate of any disability group (U.S. Department of Education, 2001) (NIMH currently)

14 Chautauqua Tapestry Why Tapestry? Quick history Goals Partners

15 SAMHSA s Theory of Change

16 Expansion = System Change System of care expansion is not a project Goal is sustainable systemic changes in child-serving systems to improve services and outcomes Occurs with or without federal grant Grants are venture capital to achieve larger system change Linked to other system reforms

17 Keys to Successful Expansion 1. Commitment of highlevel policy and decision makers 2. Realistic goals 3. High-priority goals 4. Specific, concrete strategies 5. Strong leadership to manage implementation 6. Linkages with larger system reforms in environment 7. Cross-agency partnerships 8. Commitment across key stakeholders 9. Staff and resources allocated to implementation work

18 Evaluation as Sustainability Strategy Similar organizations collecting quality data, but uncoordinated and isolated Similar organizations collecting quality data for the same outcomes in a coordinated and collaborative network Similar organizations collecting quality data that align under similar outcomes for broad community impact

19 Collaboration Defined: A mutually beneficial relationship between two or more parties who work toward common goals by sharing responsibility, authority and accountability for achieving results.

20 New York State

21 Evaluation Examples Jamestown Public Schools North Rockland School District Manchester City Council, UK

22 Critical Realism and Outcomes Sayer (1984): Realism links theoretical abstraction with empirical research Essentially eclectic, the realist approach seeks to move away from what may be now rather sterile epistemological debates surrounding scientific versus pluralistic and constructivist paradigms, using all methodologies as appropriate within the realist framework. The approach facilitates the interrogation of empirical data with a view to identifying the nature of specific structural influences on outcomes (p.45). Realist evaluation embraces outcome measurement and also investigates the factors that influence the outcomes

23 Realist Evaluation Realist evaluation seeks to evaluate practice within the realities of society, based on Emergent and Symbiotic elements of natural law. All systems are emergent and constantly in a state of evolution, along with the reality that we are all symbiotically connected to nature and to each other. Practice takes place in an open system that consists of a constellation of inter-connected structures, mechanisms and contexts. Realism aims to address all the significant variables through a realist effectiveness cycle which links the models of intervention with the circumstances in which practice takes place (Rom Harre s models, analogous with reality) Explanation at any one time requires further investigation and further explanation (continuous testing & development of Rom Harre s models)

24 Purposes and Objectives of evaluation Research methods drawn from both epidemiology and efficacy research traditions in a realist evaluation Partnership with human service agencies & schools to investigate what works and for whom. Real live data mined from human service agencies is used to investigate the effectiveness of the human service interventions. Emphasis is on data naturally drawn from practice. Binary logistic regression as part of epidemiologic evidence based on association, environmental equivalence, and population equivalence. How evaluators and agencies can make the best use of the available data to inform practice.

25 Purposes and Objectives of evaluation Research indicates that the reform of the system of care and the use of initiatives such as wraparound are effective in improving mental health and functioning in school (Kutash, 2006; Reback, 2010; Goldenson, 2011) However, most studies have focused on at risk groups rather than the total school populations. The purpose of this study is to undertake a 100% evaluation of all school-based services, utilizing data on the entire school populations, in a longitudinal study.

26 Evaluation Strategy With Each Agency Evaluation resource and service for you and for each participating agency How to access and to use your own MIS data How to analyze this data repeatedly to inform practice Carried out with you and for you only you decide who to share with and how to use the findings

27 Realist Evaluation: What Interventions work & in what circumstances A combination of efficacy research & epidemiology traditions Management Information System (MIS) Data routinely collected but typically not used for evaluation in agencies Investigate interrelationships between outcomes, client demographics, client circumstances, & services provided (interventions) Methods such as binary logistic regression can predict the likelihood of effectiveness of an intervention in given circumstances Use findings at regular intervals to better target and develop services

28 Education Partners in SOC Education plays a critical role in the development of children.. Positive learning experiences help prevent emotional and behavioral problems.

29 School Examples of Realist Evaluation 17 NYS school districts have participated 100% data from school + participated agencies What works and for whom in achieving school, agency and system of care outcomes Examples of school data Demographics Intervention Outcome Ethnicity Gender Lunch status, Special education School based interventions Pre-school programs Mental health DSS GPA/ State tests Behavior incidents Attendance Drop out rates Graduation rates

30 Data analysis and utilization Single system & one group pre/posttest Comparison of outcomes between baseline and subsequent periods and those receiving and not receiving interventions Investigation of patterns between outcomes, demographics and interventions Binary logistic regression Data analysis in partnership with schools and agencies Utilization of findings to develop and improve services for children and families

31 Outcomes: Jamestown Public Schools Strengths and Difficulties Questionnaire (SDQ) April 2015 and April 2016 Recorded Incidents of Behaviour 2014/15 and 2015/16 North West Evaluation Association (NWEA) Measures of Academic Progress (MAP) Fall 2015, Spring 2016 & Fall 2016 (grade levels K to 10)

32 Universal Screening Tool: SDQ Strengths and Difficulties Questionnaire (SDQ) Brief measure of pro-social behaviour and psychopathology, 3-17 yr olds Goodman (2001) reliability.73 Five factors: emotional symptoms, conduct problems, hyperactivity, peer relationships and pro-social Grade levels K to 4 in April 2012, 2013, 2014, 2015 & 2016

33 SDQ 2012 Normal Scores/Ethnicity

34 SDQ 2012 Predictors for abnormal/borderline total scores

35 Predictors for Change from 2012 to 2013 Total SDQ

36 Hyperactivity Level 2015 SDQ

37 Hyperactivity Level 2016 SDQ

38 Hyperactivity Level Change from 2015 to 2016 SDQ

39 Hyperactivity Level Change from 2015 to 2016 SDQ: Predictors for improvement

40 Hyperactivity Level Change from 2015 to 2016 SDQ: Striders Advocacy

41 Hyperactivity Level Change from 2015 to 2016 SDQ: Jamestown Behavioral Health Clinic

42 Tapestry Partners Again Predictors for Improved Behavior in Schools Recorded incidents of behavior Compared between and school years Overall mean was 0.96 in 2014/15 & 1.58 in 2015/16 recorded incidents per student ( n = 5134) in Jamestown Public Schools in both years Once again Tapestry partners services were predictors for improved behavior, this time several more!

43

44 Safari Improved Behavior

45 Improving Measurement of Academic Achievement Northwest Evaluation Association (NWEA) Measures of Academic Progress Normative Data for Reading and Mathematics More reliable than GPA Used three times a year up to grade level 10 Beginning from Baseline Year 2015/16 First two times used Rasch Unit Scale (RIT) but for the third assessment will use expected growth Of those proficient at level 3 or 4, 51.8% demonstrated growth that was less than expected in the first two reading assessments

46 NWEA Reading Fall 2015 & Spring 2016

47 NWEA Reading Fall 2015 & Spring 2016

48 NWEA Reading Fall 2015 & Spring 2016

49 NWEA Reading Fall 2015 & Spring 2016

50 NWEA Reading Fall 2015 & Spring 2016

51 NWEA Reading Fall 2015 & Spring 2016

52 NWEA Reading Fall 2015 & Spring 2016

53 NWEA Reading Fall 2015 & Spring 2016

54 NWEA Reading Fall 2015 & Spring 2016

55 NWEA Reading Fall 2015 & Spring 2016

56 NWEA Reading Fall 2015 & Spring 2016 Predictors for Improving RIT Score at Median 9 or Higher

57 Plan for Chautauqua Tapestry 2.0( ) Chautauqua County the only one to receive SAMHSA s Gold Award for Outstanding Local Evaluation. Tapestry 2.0 will deepen its evaluation with school districts and human service agencies in the county. Chautauqua County is a world-leader in realist evaluation, and to date 28 agencies (including 7 school districts) have collaborated

58 New York State

59 Rockland County MACSHY

60 Rockland County MACSHY Leaders

61

62 Rockland County PSY SY_Rafael_FINAL.mp4?dl=0

63

64 Outcomes: North Rockland Schools Recorded Incidents of Behaviour- 2016/17 (grade levels K-12 th ) Istation Universal Screener- Fall 2016 & Spring 2017 (grade levels 3 rd to 6 th ) Developmental Reading Assessment 2 (DRA2)- Fall 2016 & Spring 2017 (grade levels 1 st - 4 th ) North West Evaluation Association (NWEA) Measures of Academic Progress (MAP) Fall 2015, Spring 2016 & Fall (grade levels K to 10) from 2017/18 School Year

65 North Rockland Schools 2017/18

66 Recorded Incidents of Behavior 2016/17 (8.9%) to 2017/18 (11%; N = 8026) Mental health affects school functioning (Reback, 2010; Desrocher, 2015; Rossen & Cowan, 2014). However, most studies have focused on at risk groups rather than the total school populations. Rockland County utilizes big data continuously on entire school populations with North Rockland Schools. It was found that, out of the 85 that had received county mental health services, 15.3% had improved on the number of recorded behavior incidents as compared with 4.4% of the rest of the school Out of the 62 that had received DSS interventions, it was found that 9.7% had improved on behavior incidents as against 4.5% that had also improved in the school population not receiving these interventions. There were 16 that had been involved with Partnership for Safe Youth, and of these 4 (25%) had improved as against 4.5% of the rest that had not been involved with the PSY; however, the PSY total was less than 20 at this stage and therefore not included in the inferential statistics.

67 Recorded Incidents of Behavior 2016/17 (8.9%) to 2017/18 (11%; N = 8026) Binary Logistic Regression: Incidents Improved or not--variables in the Equation B S.E. Wald df Sig. Exp(B ) Gender2017(1) Ethnicity2017(2) Ethnicity2017(3) MHPatientOrNOTALLto (1) FreeReducedornot(1)

68 Plan for Transforming Rockland County( ) National Evaluation Requirements Local Evaluation Strategies Continuous realist evaluation of services utilizing entire Management Information Systems Department of Social Services County Mental Health Services Probation Service North Rockland and Nyack School Districts Continuous quality improvement

69 Plan for Rockland County( ) Rockland County Evaluation will expand from the largest North Rockland Schools to include all 8 school districts (n = 41,000) One of 37 in NY, Board of Cooperative Educational Services (BOCES) plays central role with Partnership for Safe Youth 100% data from County Mental Health, Department of Social Services, Probation Service and other human service agencies, and merged with whole school big data

70 Example: Manchester Youth Justice Service in England This interim report is based on electronic data comprising of a 100% sample of 2,526 youth in the 4-year period January 2011 to December % were male, and 62.2% of the youth were white The average age at date of arrest for all youth was 15 years, ranging from 10 to 18 years.

71 Manchester Youth Justice Service in England Offence database recorded every episode of an offence reported on the youth, Work out those that had offended after the first recorded offence, and those that had more than one, in the 4-year period from January 2011 to December It was found that 66.1% had not offended again after the first recorded offence (as compared with 45.6% in the previous 10-year analysis reported at AEA 2012).

72 Binary Logistic Regression Model

73 Predictors for returning Age at first order. Those older by every year were 1.5 times more likely not to return Gender. Females were more likely not to return than males. Those who had completed the order were 7 times more likely not to return. Youth living with parents or relatives were 2.6 times more likely not to return. Those with higher baseline ASSET scores were more likely to return Those on youth caution more likely not to come back

74 Manchester Youth Justice Service: Linking Data with Practice Using our data differently Clearer picture of our cohort Understanding what is working and what is not working Helping to target resources more effectively Identifying predictors of re-offending

75 Manchester Youth Justice Service: Looking Forward Improving the way we record data Working with partners to improve use of data Merging this data with a wider pool of data within the City of Manchester e.g. schools, police, health, benefits agency Targeting resources more effectively to get better outcomes

76 Conclusion Realist evaluation strategies help to utilise the available whole school data and to conduct deeper analysis Explanations for differences between those that improve on an outcome and those that do not Identifies patterns in the data where multiple factors are influencing the outcome, and selects the main factor or factors responsible for the outcome, with a prediction of the odds of achieving a given outcome in particular circumstances Repeatedly investigate changes in outcomes, intervention & the contexts