IMPROVING MENTAL HEALTH: A DATA DRIVEN APPROACH

Transcription

1 IMPROVING MENTAL HEALTH: A DATA DRIVEN APPROACH A LOOK INTO OUR CHILD POPULATION Winter Statistics and Research: Darren Lubbers, Ph.D. Brittany Shaffer, M.S. The Child and Adolescent Functional Assessment Scale (CAFAS) is used state-wide to assess the clinical progress of our child consumers. This report walks through several analyses done at Ottawa County CMH involving CAFAS data. In the following analyses we investigate: (1) When gains occur during treatment, (2) What factors affect gains, (3) Strengthening the criteria for clinical significance, and (4) The agreement between categorical discharge type selected in agency discharge summaries and the change in total CAFAS score. All analyses presented in this report can be replicated at other CMH agencies throughout the state due to the fact that we are all using the same standardized tool the CAFAS. We begin by taking a look at consumer progress, and when gains are occurring. Methods CAFAS data was collected for consumers who had an initial assessment, three month assessment, six month, and nine month assessment completed by the same assessor (exception: initial assessment). This resulted in a sample of 38 children. Progress was assessed by looking at the average total CAFAS score at intake, 3 months, 6 months and 9 months for the same 38 children. Maine s analysis uses 3 independent groups of children (you will notice 4 different sample sizes for Maine at intake, 3 months, 6 months, 8 months, and 12 months). Masters Degree Research Associate Intern: Colleen MacCallum Executive Director: Lynne Doyle, MPA, MA Family Services Director: Ann Heerde, LMSW What did Maine and Ottawa Find? Table 1 shows a summary of the progress seen in Maine over the first year of treatment, and Table 2 shows a summary of the progress seen in Ottawa over the first 9 months of treatment. We see both agencies reach peak gains between 6-8 months. After this point, both agencies see an increase in average total score, meaning they have lost some of their gains. Table 1: Progress Observed in Maine PROGRESS OBSERVED IN MAINE n Average Total Initial Month Total Month Total Month Total Month Total Overall Gains: 16.6 points Table 2: Progress Observed in Ottawa PROGRESS OBSERVED IN OTTAWA n Average Total Initial Month Total 6 Month Total 9 Month Total Overall Gains: 18.0 points Community Mental Health of Ottawa County James Street Holland, MI

2 When are Gains Occurring? Clinical Significance The clinical significance of these findings is two-fold. First, these findings suggest Ottawa County CMH is similar to another state agency in the population it serves, as well as the gains made within that population. Both CMH agencies have children entering services, on average, with an initial total CAFAS score of roughly points. This sample suggests Maine and Ottawa CMHs are also making similar gains within their child populations (16.6 versus 18.0, respectively). Secondly, these findings suggest there may be a point of diminishing returns in treating children. Both agencies found after 8 months of treatment, gains start to be lost. This suggests for some children it may actually be detrimental to stay in services for too long. What qualifies as too long is debatable, and should be studied further in future analyses. The samples from these two agencies suggest sometime between 6 and 9 months of treatment is optimal. After this point, both agencies saw a loss of gains within their sample. This interval of 6 to 9 months could be used in future analyses as a starting-point for the investigation of optimal length of stay. Do Gains Depend on Initial? The following analyses were completed using analysis of covariance (ANCOVA). Let s take a moment to set clinical examples aside and look at bowling. Consider a novice bowler who averages 80 points/game. With lessons and regular practice, this bowler could easily increase his/her score by points. Now let s consider a professional bowler who averages 280 points/game. Regardless of how many hours this bowler spends practicing and perfecting his/her craft, it will be impossible to increase his/her score by points because a perfect game is 300 points. In this situation, the initial average score is the covariate. It affects the possible gains made by the bowler. Initial score needs to be accounted for if gains made by the novice and professional bowler are to be compared. Performing an ANCOVA will control for this initial points/game. Now considering CAFAS data again, we must use this concept in comparing gains made by children who enter services with different initial total CAFAS scores. 2

3 Factors Affecting Gains Winter Gains Made by Our Child Consumers: Do they differ by gender and diagnosis? In the following analyses, we will assess whether the gains made by child consumers differ by diagnostic group and/or by gender. Through use of the CAFAS score, we can quantify the gains made by looking at the difference in CAFAS score from initial to exit total CAFAS score. Average Gains Made by Gender and Diagnostic Group within the Sample Table 3 provides the average gains made by gender within each diagnostic category. For attention-deficit and anxiety disorders, it appears there is little difference in gains between genders. For the other 3 diagnostic categories, however, greater gains were observed for females in the sample. Table 3: Average Gains Made by Gender within Diagnostic Categories Diagnosis Gender n Average Gains ANXIETY DISORDER Female Male ATTENTION-DEFICIT DISORDER Female Male CONDUCT DISORDER Female Male MOOD DISORDER Female Male OTHER DISORDER Female Male Are these differences occurring by chance in the sample? The information presented in Table 3 suggests there may be differences in the average gains made based on gender and diagnostic category. When initial total CAFAS score was taken into account through analysis of covariance (ANCOVA), however, it was determined there is only evidence that gender is affecting gains made. Which gender is making greater gains? The adjusted average gains were compared for each gender and it is estimated that female child consumers, on average, are making roughly 9 point greater gains than their male counterparts. Clinical Significance These findings suggest some aspect of our services better fit the needs of female consumers. The ANCOVA analysis above tells us female child consumers on average are improving by 9 more points than male child consumers on average when initial total CAFAS score is controlled. Further analysis should be done to investigate what aspects of our services may be contributing to this difference in average gains made. 3

4 Factors Affecting Gains Gains for our Child Consumers: Do they differ by treatment intensity? In the following analyses, we investigate whether or not equal gains are being made in different treatment intensities. The differing levels of treatment intensity investigated in this analysis are outpatient and home-based. Average Gains Made by Treatment Intensity Table 4 provides the average gains made by children in each treatment-intensity. On average, children in home-based care are making 40 point gains; that is, their CAFAS total score is decreasing 40 points, on average, during treatment. Children in outpatient care, on average, are making point gains; that is, their CAFAS total score is decreasing, on average, points during treatment. It is important to note, however, Table 4 also shows children are entering home-based care with an average initial total CAFAS score of , while those entering outpatient have an average initial total score of To account for these difference, analysis of covariance (ANCOVA) is used. When controlling for initial total score, there is not statistically significant evidence that average gains differ due to treatment intensity. Table 4: Looking at Average Gains AND Average Initial Total CAFAS Treatment Intensity n Average Gains Average Initial Total CAFAS Home-based Outpatient Clinical Significance The clinical significance of these findings is three-fold. First, and perhaps most importantly, it is clinically significant that home-based treatment and outpatient treatment are both resulting in positive consumer outcomes. The children who are entering outpatient treatment with a total initial score of and reduce their total score by points are making just as much progress as the children who enter home-based care with an initial total score of and reduce their total score by points. Both treatment intensities are creating positive outcomes for our consumers. Secondly, these findings challenge the idea that a 20 point gain in total CAFAS score is clinically significant for all children. If a decrease from an initial total score of is equivalent to a decrease from an initial total score of , then clinically significant gains cannot and should not be defined as a single value for all children. The value that denotes a clinically significant gain is dependent upon the consumer s initial total CAFAS score. Thirdly, this analysis highlights the importance of using appropriate analyses. This analysis could have been performed without controlling for initial total CAFAS score as a one-way ANOVA with treatment intensity as the sole factor of interest. If we had performed the analyses ignoring initial total CAFAS score, we would have ended up estimating that, on average, home-based treatment results in point greater gains for child consumers. That is a huge misrepresentation of the progress being made because it does not take into account the different severities being treated by home-based and outpatient treatments. This highlights the importance of using correct methodologies, and it showcases how easily data can be misrepresented. 4

5 Factors Affecting Gains Winter Gains in Children: What service data is useful in predicting gains made by children? Predictive models can be used to predict how a child will respond to treatment, or to determine how certain variables affect gains made by a child. This analysis outlines the process used to fit a multiple linear regression model which uses gender, total initial CAFAS score, and number of crisis units to predict a child s gains (change in total CAFAS score). Fitting a Linear Model All possible predictor variables (age, gender, program, units of service (by type), total cost, and initial CAFAS scores) were tested in the model using SAS 9.3 to predict average gains for child consumers. The final multiple linear regression model included: gender, initial total CAFAS score, and the number of crisis units received. Figure 1 shows the corresponding path model illustrating this relationship. The Predictive Equation The resulting prediction equation is: Gains = (gender = female) (initial total CAFAS score) (No. of crisis units) As we saw in previous analyses, gains depend on gender and initial total CAFAS score. This prediction equation tells us in addition to gender and initial score, the number of crisis units received is also useful in predicting average gains. For each crisis unit received, we expect average gains to decrease by 3.6 points. Clinical Significance What this analysis highlights is the negative correlation between crisis units received and average gains made. This analysis shows for each crisis unit received, our children s average gains decrease by points. It also shows there is not a linear relationship between service units and gains made, meaning we cannot predict gains based on the current service data we are collecting. Figure 1: Corresponding Path Model 5

6 Strengthening Clinical Significance Gains in Children: When are gains clinically meaningful? Currently, a 20 point decrease in total CAFAS score is being used as the gold-standard for a clinically meaningful improvement in our child consumers. This standard has been stated by Hodges, Xue, and Wotring (2004) as well as Manteuffel, Stephens, and Santiago (2002). Manteuffel, Stephens, and Santiago (2002) stated, a 20 point change in the CAFAS total score was defined as clinically significant. This translates into a 0.44 standard deviation. Hodges, Xue, and Wotring (2002) further explain why they believe roughly half a standard deviation is useful in defining a clinically significant improvement by saying, Requiring a change of one-half standard deviation seems appropriate given this would correspond to a moderate effect size, as defined by Cohen (1988). Based on Cohen s rule of thumb, a moderate effect size is one-half standard deviation. Since we have found evidence that gains depend on initial total CAFAS score, however, it may be useful to adjust this number based on initial score. Calculating Clinical Significance There are several ways to calculate clinical significance; two of the most common are effect size and percent change. One effect-size method is the Gulliksen-Lord-Novick method. This method accounts for the fact that gains depend on entrance score. Unfortunately, this method also depends on internal consistency of the tool itself. Internal consistency is measured by Cronbach s alpha, and in the case of the CAFAS, Cronbach s alpha is between 0.63 and 0.68 (Bates, 2001). This means internal consistency with the CAFAS is acceptable, but not great. The Gulliksen-Lord-Novick method penalizes for low internal consistency scores by raising the number of points a child s total score must decrease for clinical significance. That being the case, this method did not provide reasonable minimum gains for clinical significance. As an alternative, we have used a combination of Cohen s moderate effect size and percent change. Cohen s moderate effect size rule-of-thumb when applied to our data says that a 17.3 point decrease in CAFAS score is a clinically significant gain for our average initial score or This 17.3 point decrease in an initial score of 79.6 is a 22% change in initial score. For this reason, a 22% change was applied to all possible entrance scores. These values can be seen in Table 5 to the right. This percent change also corresponds to the percent change used by the Gulliksen-Lord-Novick method when internal consistency is good (Cronbach s alpha = 0.78). With the penalty for the CAFAS internal consistency, Gulliksen-Lord-Novick says a clinically significant gain is a 35% change in initial score. The adjusted method lowers this to a 22% change. We then used the same methodology to calculate a small and large effect for our average initial CAFAS score of Cohen defines a small and large effect size as a change in the standard deviation by 0.2 and 0.8, respectively (Cohen, 1992). When applied to our data, a 6.9 point decrease in an initial CAFAS score of 79.6 is considered a small gain as defined by Cohen s small effect size. This 6.9 point decrease in the average initial CAFAS score is a 9% change in initial score. Therefore, a 9% change was then applied to all possible entrance scores. Similarly, a 27.4 point decrease in an initial CAFAS score of 79.6 is considered a large gain resulting in a 35% decrease in the initial CAFAS score. Therefore, a 35% change was then applied to all possible entrance scores. Table 5, on the next page, summarizes the minimum decrease in CAFAS score required by each possible intake score in order to classify a gain as small, moderate, or large. 6

7 Strengthening Clinical Significance Table 5: Clinically Meaningful Gains by Initial Initial Total Adjusted Meaningful Gains Initial Total Adjusted Meaningful Gains Small Moderate Large Small Moderate Large Based on previous research, we know that approximately 66% of our clients show improvement in their CAFAS score. Figure 2 below, also shows this as each bar approaches the 70% mark. From the figure below, we see that the outcomes from CMH treatment and the combination of CMH and contractual treatment exceed the outcomes of those just receiving treatment through a contractual provider. Interestingly, we found that the majority of gains made in all programs were considered large indicating these clients achieved a minimum change of 35% in their CAFAS score. Figure 2: Clinically Meaningful Gains by Program 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Magnitude of CAFAS Gains Achieved CMH of Ottawa County Any Gain Small Gain Moderate Gain Large Gain 3% 13% 11% 43% 14% 5% 52% 13% 9% 29% 2% 13% 10% 38% CMH CMH/CONTRACTUAL CONTRACTUAL OVERALL 7

8 Discharge Type: CAFAS vs. Categorical Positivity of Discharge as Determined by CAFAS s vs. Categorical Discharge Type Tracking consumer progress can facilitate improvement of treatment and consumer outcomes. Currently, there are two measures used to track progress of child consumers at CMHOC. These two measures are the CAFAS score and the categorical discharge type given by the clinician at the time of discharge. This study examines the agreement between these two measures at the time of discharge. Progress According to Total CAFAS s A frequency table was generated for positivity of discharge according to the difference in total CAFAS score. These frequencies can be seen in the summary provided in Table 6. Of the 256 children, 66.02% (n = 169) improved during treatment, 11.72% (n = 30) saw no change, and 22.27% (n = 57) declined during treatment. Table 6: Positivity as Indicated by Change in CAFAS (Initial to Exit) According to CAFAS s Positivity Frequency Percent Positive Neutral Negative Progress According to Categorical Discharge Type The same frequency procedure was performed according to categorical discharge type. These frequencies can be seen in the summary provided in Table 7. Using categorical discharge type to determine positivity, 32.03% (n = 82) of the 256 children had a positive discharge, 35.94% (n = 92) had a neutral discharge, and 32.03% (n = 82) had a negative discharge. Table 7: Positivity as Indicated by Categorical Discharge Type According to Categorical Discharge Type Positivity Frequency Percent Positive Neutral Negative

9 Discharge Type: CAFAS vs. Categorical Clinical Significance These findings suggest categorical discharge type is not a fair indicator of consumer progress. Using discharge type in performance reporting may result in the underreporting of positive consumer outcomes. This is particularly important for programs that do not currently have a CAFAS equivalent. For those programs, the categorical discharge type may be the only variable available on which to base program success. Figure 3 below displays how the positivity reported differs based on measurement tool. Many positive gains are being categorized as neutral or negative discharges. Of the consumers who were reported (through categorical discharge type) to have had a negative discharge, 56.10% actually saw a decrease in total CAFAS score. Of the consumers who were reported (through categorical discharge type) to have had a neutral discharge, 58.70% saw a decrease in total CAFAS score, 32.61% saw an increase in total CAFAS score, and only 8.70% saw no change in total CAFAS score. These findings suggest categorical discharge type is not an accurate measure of consumer progress and an alternative measure may be more appropriate for performance reporting. Figure 3: Positivity as Indicated by Total CAFAS v. Categorical Discharge Type 9

10 10

11 References Cohen J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), Hodges, K., Xue, Y., & Wotring, J. (2004). Use of the CAFAS to evaluate outcome for youths with severe emotional disturbance served by public mental health. Journal of Child and Family Studies, 13(1), Manteuffel, B., Stephens, R. L., & Santiago, R. (2002). Overview of the national evaluation of the comprehensive community mental health services for children and their families program and summary of current findings. Children's Services: Social Policy, Research, and Practice, 5(1),