How to continuously evaluate state-wide systems of care utilizing 100% sample data from all schools, mental health and other services Paper Presented at 27 th. Annual Children s Mental Health Research and Policy Conference Tampa, Fl, March 2-5, 2014 Mansoor A. F. Kazi, PhD, University at Albany, State University of New York Patricia A. Brinkman, Director Community Mental Hygiene Services, Chautauqua County, NY Jon Anderson, Chautauqua Tapestry, New York Mansoor A. F. Kazi, Assistant professor Behavioral Health, School of Social Welfare, University at Albany mkazi@albany.edu American Evaluation Association (eval.org) Co-Chair and founder of the Social Work Topical Interest Group. New York Success: Connecting Systems of Care With Children and Families STAKES/THL Realist Evaluation Partnerships Tapestry of Chautauqua County, New York SAMHSA s Gold Award for Outstanding Local Evaluation 2010 Training Institute Based on Kazi, M. A. F. (2003) Realist Evaluation in Practice, London: Sage 1
Chautauqua Tapestry Family driven ~ Youth guided ~ Culturally sensitive Community based ~ Evidence-based What is Chautauqua Tapestry? An opportunity and challenge for Chautauqua County service and support providers to partner with youth who have emotional and behavioral challenges and with their families to create an accessible, responsive, appropriate and effective service delivery system. Chautauqua Tapestry brings together providers, families and the community to share responsibility and resources to enhance the delivery of services and improve the lives of youth and families. 2
Seeking Collaborative Partners Education Mental Health Public Health Child Welfare Family Court New York Success: Connecting Systems of Care With Children and Families 3
NYS System of Care Expansion - Background 2010 CLMHD Children & Families Committee Position Paper & SOC Briefing Paper (Levison-Johnson) 2011 Funding through NYS OMH administered through the CLMHD Children and Family Committee to begin Learning Collaborative Established in 2011 with Twelve Learning Communities in Phase 1 and Nine Mentor communities 2011 2013 Awarded 10 mini-grants ($5,000 each) to Phase 1 to support SOC Expansion Proposals required how community will promote the principles of SOC Three meaningful & manageable outcomes to measure impact Identify key partners 1:1 Consultation and Coaching All day Learning Sessions January 31, 2012 and January 15, 2013 Monthly webinars Started February 2012 What is NYS Success? In 2012, SAMHSA awarded Upstate New York with a four year, four million dollar grant to support broad-scale operation, expansion and integration of systems of care through the creation of sustainable infrastructure. Strengthening Family and Youth involvement County State Structure, Communication and Planning Training & TA Pool to build capacity Innovation Fund to invest in cross-system pilot activities Strategic Financing Plan (guidance on flexible funding of supports) Evaluation 4
Seneca 3/7/2014 Systems of Care Statewide vs. a Statewide System of Care What s Different? Active collaboration and communication between state representatives and localities A different way of working together to effect policy & regulatory reform that supports system of care initiatives Facilitating intentional partnerships focused on maximizing benefits by sharing strengths Building systems of care infrastructure from local/grassroots up to state leadership Franklin Clinton Saint Lawrence Jefferson Essex Lewis Hamilton Warren Oswego Washington Niagara Orleans Genesee Monroe Wayne Onondaga Oneida Herkimer Fulton Saratoga Erie Wyoming Livingston Ontario Yates Cayuga Cortland Madison Otsego Montgomery Schenectady Schoharie Albany Rensselaer Tompkins Chenango Schuyler Chautauqua Cattaraugus Allegany Steuben Chemung Tioga Broome Delaware Greene Columbia Ulster Sullivan Dutchess Orange Putnam Westchester Rockland Bronx New York Queens Nassau Suffolk Richmond Kings 5
Evaluation: Main Objectives Expand system of care in a state-wide approach to the development of a comprehensive array of services and supports for youth and families How to evaluate the effectiveness of changes in the services as the system of care is expanded How to utilize data dumps from the management information systems of schools, mental health and other services to continuously evaluate How to use research methods drawn from epidemiology and effectiveness research (e.g. quasi-experimental designs and logistic regression) to investigate patterns between the demographics, interventions and outcomes How to use realist evaluation at regular intervals in a continuous evaluation that is sustainable How to enhance the utility of evaluation from a family perspective that places real meanings to the quantitative findings across thousands of service users How to use this data to promote cultural competence, family and youth partnerships, and interagency collaboration. Transformation Accountability (TRAC) implement TRAC National Outcome Measures (NOMS) Focus on children and youth with serious emotional disturbances Receiving home and community based waiver services Use as part of admission process (baseline) and then every 180 days or until discharge 6
Infrastructure survey Survey based on SAMHSA requirements as first stated in the Request for Applications Definitions based on current SAMHSA guidance Only one survey required from each county at the end of each quarter Local Evaluation Strategy for Each County Evaluation resource and service for you and for each participating agency How to access and to use your own MIS data How to analyze this data repeatedly to inform practice Carried out with you and for you only you decide who to share with and how to use the findings 7
Realist Evaluation: What Interventions work & in what circumstances A combination of efficacy research & epidemiology traditions Management Information System (MIS) Data routinely collected but typically not used for evaluation in agencies Investigate interrelationships between outcomes, client demographics, client circumstances, & services provided (interventions) Methods such as binary logistic regression can predict the likelihood of effectiveness of an intervention in given circumstances Use findings at regular intervals to better target and develop services Evaluation Strategy examples Tompkins County data-dump from mental health since 1980 Essex and Franklin counties: Merger with DSS, mental health since 2002 Chemung, Schuyler, Cortland, Chenango, Greene, Columbia and Ulster counties: merger with mental health, not-for-profits, DSS and school district data Rensselaer County Merger with probation, mental health, Head Start and Troy Schools Started with Broome, Cayuga, Yates and other phase 2 counties 8
Tapestry of Chautauqua County, New York Jamestown Public Schools: Universal Screening, and Evaluating Services With School Outcomes Mansoor A. F. Kazi, PhD, University at Buffalo (The SUNY) Tina Sandstrom, Jamestown Public Schools Carol Wright & Briana Palmer, Chautauqua County Mental Health School Examples of Realist Evaluation 13 School Districts (Chautauqua County); Averill Park & Troy School District (Rensselaer County);Cairo/Durham & Catskill (Greene county); Hudson schools (Columbia County) 2008/09 baseline and comparisons with 2009/10, 2010/11 and each marking period in the current year Outcomes: average school grades, state tests, discipline, attendance, drop out rates Demographics: ethnicity, gender, lunch status, special educational needs, etc. Interventions: school based interventions, summer program, mental health and other services 100% school data plus 100% agency data from participating agencies What works and for whom in achieving school, agency and system of care outcomes 9
Data analysis and utilization Single system design with each youth and one group pretest posttest design repeated at every marking period Comparison of outcomes between baseline and subsequent periods Comparisons between those receiving and not receiving interventions Investigation of patterns between outcomes, demographics and interventions Binary logistic regression to identify predictors at every marking period Data analysis carried out in partnership with schools and agencies Utilization of evaluation findings to develop and improve services for children and families at regular intervals Universal Screening: SDQ Strengths and Difficulties Questionnaire (SDQ) Brief measure of pro-social behavior and psychopathology, 3-16 yr olds Goodman (2001) reliability.73 Five factors: emotional symptoms, conduct problems, hyperactivity, peer relationships and pro-social Grade levels K to 4 in 2012 & again in 2013 10
SDQ 2012 and 2013 Scores SDQ 2012 and 2013 Scores 11
SDQ 2012 and 2013 Scores SDQ 2012 Normal Scores/Ethnicity 12
SDQ 2012 Normal Scores/IEP SDQ 2012 Predictors for abnormal/borderline total scores 13
SDQ Change from 2012 to 2013: Total Scores SDQ Change from 2012 to 2013: Total Scores 14
Jamestown County Mental Health Distribution of 2012 Total SDQ Jamestown County Mental Health Change from 2012 to 2013 Total SDQ 15
Jamestown County Mental Health Change from 2012 to 2013 Total SDQ Striders Elementary Change from 2012 to 2013 Total SDQ 16
Predictors for Change from 2012 to 2013 Total SDQ Conclusions for SDQ Screening Main contributors to SDQ total score status were hyperactivity and conduct subscales Predictors for abnormal/borderline SDQ total scores were IEP, lunch status, gender, father s level of education Significant improvements with Jamestown County Mental Health intervention Predictors for SDQ total score improvement were Striders intervention and baseline abnormal/borderline scores 17
Evaluation Strategy We are here to help you as an evaluation resource Implement TRAC NOMS and infrastructure survey requirements for SAMHSA Help you to use your own data for evaluation Can connect to others in Chautauqua County and others who have done this (e.g. data-dumps) Data findings can inform your decision-making and help make your services more effective This evaluation will also help you to be in a better position to apply for grants and other funding Enhance your own capacity for evaluation Utilizing SAMHSA funding, a free evaluation service for all participating counties (21 counties; more next year) 18