Chapter 4. Symptoms and Functioning Severity Scale (SFSS)

Chapter 4 Symptoms and Functioning Severity Scale (SFSS) Background Purpose The Symptoms and Functioning Severity Scale (SFSS) serves as the core indicator of treatment progress in the PTPB. It is a general measure of youths emotional and behavior problems intended to measure change over time in closely timed repeated measurements (e.g., two weeks apart). The SFSS is designed to provide systematic feedback on the youth s global level of severity in regard to symptoms and functioning. The SFSS is comparable to other existing clinical outcome measures but has the advantage of being short (it takes only five to seven minutes to complete) allowing for frequent assessment over time. Theory The emphasis of this battery is on the assessment of both treatment process and treatment progress. Treatment progress is to be assessed frequently so that adjustments to the treatment plan can be made while treatment is still ongoing. But, what is understood as treatment progress? In his popular book, Authentic Happiness, Martin Seligman (2002) proposes that if we consider individual well-being to range from -10 to 10 on a scale, then you can either focus to move a person from -2 to 0 or you could try to move a person from 0 to +2. The former could be described as a reduction of problems such as typical mental health symptoms (e.g., difficulty controlling oneself, feelings of depression, experience of anxiety) and the latter as an increase in happiness and hope. In the PTPB we included measures that allow the user to assess progress on both sides of that scale. The SFSS described in this chapter is a measure assessing progress in the reduction of symptom severity and increase of functionality. Thus, it is intended to assess the progress from a high severity level at the beginning of treatment, often referred to as the clinical range, into a comparably normal low level of severity. The SFSS is not too different from existing youth mental health outcome measures. Probably the most popular of these is the Child Behavior Checklist (CBCL; Achenbach, 1991) and the parallel versions of the Youth Self Report (YSR; Achenbach, 1991) and the Teacher Rating Form (TRF; Achenbach, 1991). Among other things, the Achenbach scales are popular because they provides a total problem score plus eight subscale scores (e.g., anxious/depression and attention problems). However, it requires the respondents to answer 118 questions. Completing this questionnaire can easily take the complete clinical session and is, thus, not a feasible option for frequent administration. In contrast, the SFSS provides only two subscale scores (internalizing and externalizing) in addition to the total score, and uses only 33 items, or less if one of the shorter versions is used. 30

The 33 items cover behaviors and experienced emotions that are linked to the most typical mental health disorders for youths including ADHD, conduct/oppositional disorder, depression and anxiety. In addition, the SFSS includes items related to peer and family relationship problems. However, the SFSS is not intended as a diagnostic instrument. It provides a global indicator for the overall reduction in symptom severity and increase in functionality. If a more targeted assessment is desired, we recommend administering a specialized scale on a less frequent basis (e.g., every three months) in addition to the SFSS. Since many youths experience co-morbid conditions, a global indicator covering a range of symptoms has important advantages in monitoring treatment progress. We recommend using all three versions (youth, adult caregiver, and clinician) of the SFSS simultaneously. There are significant differences in the perception of the problem by the different stakeholders in the treatment process. In a meta-analysis by Achenbach and his colleagues (Achenbach, McConaughy, & Howell, 1987), for example, they found that while the agreement between similar informants (e.g., two parents) is relatively high (mean r = 0.60), the correlation between different types of informants (e.g., parent and teacher) is low (0.28) and even lower if the scores of the child or youth are compared to those of others (0.22). A more recent review by Meyer and colleagues (2001) generally confirms these findings. Consequently, it seems advantageous if not necessary to triangulate these three perspectives. The brevity and the simplicity of the corresponding SFSS versions facilitate this type of comprehensive assessment. History of Development The SFSS has a long history of development. An earlier form of the Symptoms and Functioning measure was included in the Child and Adolescent Measurement System (CAMS; Doucette & Bickman, 2001) which was used in several child and adolescent mental health research projects. Experience with this earlier form of the scale led to several revisions. One of the major revisions we undertook was to create a better balance among externalizing and internalizing items. Using the American Psychological Association s Diagnostic and Statistical Manual of Mental Disorders Text Revision (DSM-IV-TR; 2000) as a general orientation, we made sure to include items that cover the main disorders described for youths: ADHD, conduct/oppositional disorder, depression and anxiety. In addition, the SFSS includes items related to peer and family relationship problems. The revised form was then reviewed by the clinical leadership of two mental health provider organizations. We also conducted focus groups with several clinicians at a provider organization in Nebraska. In addition we completed three pilot tests at three different provider organizations (the provider in Nebraska, a provider of home-based clinical services in Tennessee and Florida, and a crisis center in New York). Finally, we conducted cognitive interviews with youths and their caregivers. These reviews and tests led to more revisions that resulted in the 33-item form of the SFSS (SFSS-33) tested in the current psychometric study of which the results are presented in this chapter. In addition to the SFSS-33, we also evaluated two parallel shorter forms of the SFSS each containing half of the items. In order to create these shorter forms, we matched items on 31

their content area (e.g., ADHD) and their measure score (the item difficulty level). Then, one half of the matched pair was included in the SFSS Short Form A and the other half in the SFSS Short Form B. The idea of these short forms is to have two very short measures assessing the same construct (severity level) at a comparable difficulty level so that the total scores of these two measures can be compared across administrations. The type of information each single item represents will be covered over the course of two measurement instances, while the same total score can be calculated for each measurement instance. Testing and analyses in addition to the ones presented in this chapter are planned in regard to the construct and predictive validity and the sensitivity to change of all forms. Structure The SFSS has three forms of different length for each respondent group: adult caregiver, clinician, and youth. The three forms are a 33 item form (SFSS-33), and two equivalent short forms (Form A and Form B). The SFSS-33 includes a question about self-harm. This self-harm question often requires immediate action by the clinician, which sometimes is difficult to do if the SFSS is not processed immediately. Excluding this item does not change the psychometric properties of the scale in any significant way. Thus, the clinician has the option to simply exclude this item from the scale. The short Form A consists of 17 items while Form B includes 16 items. Each item asks the respondent to rate how frequently the youth experienced certain emotions or showed certain behaviors within the last two weeks (e.g., how often they got into trouble). The frequency is rated using a five-point Likert scale and includes the following answer categories: never, hardly ever, sometimes, often, and very often. The SFSS Total Score is calculated based on a linear transformation of the item average if at least 85 % of the items are completed. Since the average is practically identical for all different lengths of each respondent version this results in the same type of score ranging from 42 to 105 for adult caregivers and clinicians, and 32 to 107 for youths. The total score for a youth can be compared for the same youth over time as a trend and it can be compared to comparison samples, such as the psychometric sample presented in this manual. Based on these comparison samples, one can assess whether a score should be considered high, medium, or low. In addition, we matched the SFSS Total Score to those of the total problem score of the CBCL as explained further below. Clinicans who are familiar with CBCL scores may find this a useful way of interpreting the SFSS Total Scores. In addition to the total score, there are also two subscale scores available: Internalizing and Externalizing. In order to create these subscale scores, we divided the items into internalizing and externalizing symptoms, as it is commonly done in clinical practice. Internalizing disorders, such as depression and anxiety, are expressed within the individual and are focused on a clinically problematic affective and/or emotional state. Externalizing disorders, such as conduct disorder and attention deficit hyperactivity disorder (ADHD) are expressed overtly. It is important to note that these categorizations were done using theoretical criteria prior to the empirical analyses. While there is some evidence in the psychometric data that confirms the division of the items into 32

internalizing and externalizing symptoms, the results of the confirmatory factor analyses do not meet stringent scientific criteria in satisfactory ways. Therefore, these subscale scores should serve mainly as an orientation whether certain types of items (internalizing or externalizing) scored higher or lower during a given instance of measurement. For tracking progress on the reduction of the youth s severity level, we recommend using the total score. 33

Table 4.1 below shows how the items of the SFSS relate to the short forms, the subscales, and the DSM-IV-TR categories that were used to develop the scale. Table 4.1 SFSS Items in Relation to Short Forms, Subscales, and DSM-IV-TR Categories Item SFSS- DSM-IV-TR Item Subscale No. Short Form Category 1 Throw things when mad B E C 2 Eat more or less A I D 3 Feel unhappy or sad B I D 4 Get into trouble A E C 5 Have little or no energy B I D 6 Disobey adults B E C 7 Interrupted others A E I 8 Lie to get things A E C 9 Hard time controlling temper A E C/I 10 Use drugs B Neither Other 11 Worry about a lot of things A I A 12 Hard time getting along w/ others B E C 13 Threaten or bully others B E C 14 Feel worthless A I D 15 Drink alcohol A Neither Other 16 Have a hard time having fun A I D 17 Feel afraid others laugh B I A 18 Have a hard time waiting turn A E I 19 Sleep more than normally do B I D 20 Hang out with kids in trouble A E C 21 Feel nervous/shy A I A 22 Have a hard time paying attention B E I 23 Get into fights with family/friends B E C 24 Lose things needed A E I 25 Have a hard time sitting still B E I 26 Have a hard time sleeping A I A 27 Feel tense B I A 28 Cry easily B I D 29 Annoy others on purpose A E C 30 Argue with adults B E C 31 Think you don't have friends A Neither Other 32 Feel too scared to ask questions B I A 33* Think about hurting self A I D DSM-IV-TR category codes: C = conduct or opposition; D = depression; A = anxiety; I = hyperactivity or impulse; C/I = conduct or impulse. *This Item requires immediate action and may be excluded at the discretion of the clinician. Items 10, 15, and 31 were uncategorized as they could be associated with any disorder. 34

Administration The SFSS may be administered during all phases, intake through follow-up, as indicated in Table 4.2, and is to be completed by the youth. The SFSS-33 may be administered or the short Form A and B may be used alternately instead. Parallel versions are available for adult caregivers and clinicians. The suggested frequency of administration is every other week or at least once a month. Table 4.2 Administration of SFSS by Phase Intake Treatment Discharge Follow-Up Y A C Y A C Y A C Y A C Y = Youth (age 11-18); A = Adult Caregiver; C = Clinician Recommended Frequency: Every two weeks or at least once a month The suggested administration schedule of all the measures in the Peabody Treatment Progress Battery is presented in Appendix A. All PTPB measures with self-scoring forms can be found in Appendix B: Measures and Self-Scoring Forms. Description Basic Descriptives The possible range of total and subscale scores for all different forms of the SFSS is 42 to 105 for adult caregivers and clinicians, and 32 to 107 for youths. Using the well-known criteria established for the CBCL 4 the mean value of approximately 65 for the adult SFSS scales (see Table 4.3) indicates that, on average, the adult caregivers in this sample rated their children to be approximately 1.5 SDs above the mean value of 50 for a nonclinical population. Thus, on average, the youths are considered to be in the clinical range according to their caregivers. The distribution is approximately normal with no significant skewness or kurtosis. As can be seen in Table 4.6, 25% of the adult caregivers rated the severity of their children to be 73 or higher while another quarter rated the severity to be rather low with scores of 58 or less. The distributions of the 4 To make the scores comparable to a well-known clinical measure, we used linear equating to match the mean and standard deviation to that of Achenbach s popular Child Behavior Checklist (CBCL; 1991). This method is based on scores in which the mean and standard deviation of the nonclinical population are (50, 10). A score of 65, 1.5 SDs above average, is often considered as in the clinical range. This method of scaling scores goes back to the Minnesota Multiphasic Personality Inventory (MMPI) in the 1930 s and is familiar to most clinicians. The comparability of the SFSS scores to the CBCL scales was confirmed by fact that the means and standard deviations of the adult caregiver and the youth version of the SFSS are very close to those of the CBCL and YSR for the 116 cases that completed both scales (SFSS and CBCL/YSR). 35

internalizing and externalizing subscales are listed in Table 4.3 as well. As expected, the average score for the externalizing scale is higher (by almost seven points) than the internalizing average. The standard deviations of the two subscales are slightly inflated. The correlation of the two subscales scores is relatively high with r = 0.65 but not so high that they are indistinguishable. Table 4.3 Descriptive Statistics for SFSS-Adult Caregiver Summary Scores Scale N Mean Std Dev Skewness Kurtosis Min Max SFSS-33 498 65.40 11.00 0.19-0.19 43 103 SFSS Short Form A 498 65.34 11.04 0.23-0.12 43 104 SFSS Short Form B 498 65.40 11.07 0.17-0.27 43 101 Internalizing 501 63.09 11.71 0.44 0.00 42 105 Externalizing 504 70.23 14.77 0.11-0.59 42 105 The scoring of the clinician versions of the SFSS is parallel to the adult caregiver versions. As can be seen in Table 4.4, the average clinician scores in the psychometric sample were slightly lower compared to the adult caregiver ratings, suggesting that clinicians reported slightly fewer problems than did caregivers. The spread of scores as expressed by the standard deviations is also a slightly lower (about two units) than the variability in the adult caregiver scores. Again, the distribution of scores is approximately normal with no significant skewness or kurtosis. A quarter of the clinicians thought the youths to be at a high score of 69 or higher, while another 25 % at low severity of 57 or lower (see Table 4.6). The distributions of the internalizing and externalizing subscales show that the mean for the externalizing subscale is higher by about seven points. The standard deviations of the subscales are also higher compared to total score of the full scale. The correlation of the two subscales scores are lower compared to the adult caregiver and youth version with r = 0.50. That is, the two subscales share 25% of the variance. Table 4.4 Descriptive Statistics for SFSS-Clinician Summary Scores Scale N Mean Std Dev Skewness Kurtosis Min Max SFSS-33 651 62.71 8.83 0.19-0.27 43 91 SFSS Short Form A 651 62.22 9.10 0.23-0.42 43 93 SFSS Short Form B 651 63.24 8.74 0.22 0.00 43 94 Internalizing 651 60.57 10.12 0.30-0.38 42 94 Externalizing 651 67.25 11.99 0.13-0.36 42 101 Youth tended to perceive their level of severity lower as compared to their adult caregivers and clinicians. This is not surprising as we mentioned the relatively low correspondence among different types of reporters in the introduction of this chapter. Thus, the lower average ratings of the comparison sample of youth should be taken into 36

account when interpreting the SFSS scores. As can be seen in Table 4.5, the average score is between 54 and 55. It important to note that these relatively low scores imply that, on average, the youth in the psychometric sample consider themselves to be comparable to a nonclinical population of children. Only about a quarter of the youth rate themselves to be in the clinical range (see Table 4.6). Also noticeable is the higher variability in scores. Compared to the adult caregivers, the standard deviations in the youth sample are approximately 2 units higher while they are about four units higher than the clinician standard deviations. Thus, on average, youth spread their answers more across the scale range. The distribution of scores is approximately normal with low levels of skewness and kurtosis. The average score on the externalizing subscale is again higher than the internalizing one, but the difference is less pronounced than for the adult caregivers and clinicians (a difference of about four compared to seven points). The correlation of the two subscales is high with r = 0.66 but significantly less than the internal reliability estimates (see Table 4.8). Table 4.5 Descriptive Statistics for SFSS-Youth Summary Scores Scale N Mean Std Dev Skewness Kurtosis Min Max SFSS-33 612 54.60 13.10 0.44-0.18 32 101 SFSS Short Form A 611 54.39 13.09 0.45-0.33 32 99 SFSS Short Form B 611 54.85 13.22 0.45-0.09 33 100 Internalizing 589 53.74 14.94 0.61-0.30 32 99 Externalizing 588 57.91 15.45 0.41-0.17 32 107 Quartiles High scores are in the top quarter, with low scores in the bottom quarter as presented in Table 4.6. For example, for the SFSS-Adult Caregiver Total Score, a score greater than 73 is considered high, while a score less than 58 is considered low. To aid interpretation, the quartiles were used to create low, medium, and high scores and percentile ranks based on comparison to the psychometric sample. This information is presented in the last section of this chapter. Table 4.6 SFSS Quartiles Quartile Adult Caregiver Clinician Youth 100% Max 103 91 101 75% Q3 73 69 63 50% Median 65 62 54 25% Q1 58 57 45 0% Min 43 43 32 37

Evidence of Reliability Reliability Coefficients The Cronbach s alpha internal consistency reliability correlations are presented in Table 4.7. These alphas suggest a satisfactory degree of internal consistency for the total score. The short forms have, as expected, a smaller internal consistency estimate compared to the SFSS-33, as can be seen in Table 4.7. However, the difference is small and the alphas for the short forms are still in the satisfactory range, suggesting that the short forms can be used with comparable reliability. Table 4.7 Cronbach s Alphas for the SFSS Scale SFSS-33 Form A Form B SFSS-Youth 0.93 0.87 0.88 SFSS-Adult Caregiver 0.95 0.90 0.90 SFSS-Clinician 0.93 0.87 0.88 The internal consistency reliability of the two subscales is presented in Table 4.8. The reliability estimates for these subscales are also high, suggesting that the subscale scores can be used with satisfactory internal reliability based on this psychometric sample. Table 4.8 Cronbach s Alphas for the SFSS Subscales Scale Internalizing Externalizing SFSS-Adult Caregiver 0.90 0.94 SFSS-Clinician 0.89 0.93 SFSS-Youth 0.88 0.91 Comprehensive Item Psychometrics Tables 4.9 to 4.11 present the comprehensive item psychometrics. Shaded cells indicate that a criterion was out of the range of sought values, as described previously in Table 2.2 in Chapter Two. Only items with two or more shaded cells are considered problematic. Overall results for the adult caregiver, clinician, and youth versions were similar. Two items with near-floor means (alcohol and drugs) had many psychometric warnings for all three respondents. These were likely due to their high kurtosis, low variance and poor fit to the Rasch measurement model. Given the complexity of the construct and the relatively large number of items, it is not unusual to find items that do not meet all of the psychometric criteria in a satisfactory way. While they are not effective items in this 38

sample, they were retained because alcohol and drug use have content validity in a mental health index. In the youth sample, the item asking about unusual sleep pattern is also flagged as a potential problematic item. The relatively low item-total correlation (0.40) suggests that this item is not the strongest predictor of severity. However, it was determined that the deviations from the recommended criteria of quality in all three cases were so slight that the item could remain as it is. Besides using drugs and alcohol, the item about hanging out with kids in trouble is flagged as problematic in the adult caregiver sample. The high infit and outfit values as well as the relatively low discrimination value suggest that this item behaves differently from the other items in the scale and is not the most efficient item to use in differentiating youth with different severity levels. However, because of it content validity and the acceptable item-total correlation, we decided to retain the item in the scale. The item about unusual sleeping patterns is flagged again in the counselor sample. The rationale to keep this item is similar to the one for the youth version. In addition, it was considered an important goal to keep all three versions the same to have parallel versions. 39

Table 4.9 Comprehensive Item Analysis for the SFSS-Youth Item N Mean St Dev Kurtosis Worry 317 2.86 1.34-1.10 0.50 0.52 42.74 1.02 1.10 0.83 Feel unhappy 319 2.82 1.27-0.78 0.56 0.60 43.12 0.80 0.86 1.00 Temper 319 2.65 1.35-1.04 0.56 0.58 44.52 0.91 0.93 1.07 Disobey 321 2.64 1.21-0.66 0.56 0.59 44.61 0.79 0.81 1.03 Trouble 319 2.64 1.19-0.76 0.51 0.54 44.67 0.84 0.97 0.88 Paying attention 316 2.48 1.35-0.93 0.61 0.63 45.94 0.87 0.84 1.21 Eat more/less 319 2.47 1.40-1.09 0.46 0.48 46.05 1.21 1.51 0.77 Argue with adults 317 2.43 1.24-0.75 0.59 0.65 46.36 0.82 0.79 1.06 Interrupt 320 2.37 1.18-0.57 0.54 0.58 46.93 0.81 0.83 0.97 Family 319 2.36 1.27-0.66 0.64 0.70 47.00 0.77 0.75 1.17 Lose things 317 2.34 1.28-0.74 0.58 0.60 47.18 0.85 0.84 1.05 Sitting still 317 2.31 1.35-0.82 0.61 0.60 47.37 1.00 0.97 1.07 Annoy others 320 2.14 1.17-0.26 0.45 0.48 49.09 1.00 1.00 0.96 Fights with family 314 2.13 1.19-0.06 0.61 0.64 49.11 0.82 0.77 1.14 Cry easily 318 2.14 1.39-0.50 0.53 0.54 49.11 1.21 1.17 0.94 No energy 320 2.13 1.22-0.32 0.46 0.48 49.17 1.18 1.19 0.77 Nervous around others 319 2.11 1.21-0.31 0.49 0.51 49.33 1.00 1.01 0.99 Sleep 320 2.10 1.31-0.28 0.40 0.39 49.35 1.31 1.42 0.70 Lie 320 2.04 1.16-0.33 0.49 0.53 50.06 0.91 1.03 0.92 Tense 316 2.01 1.23-0.32 0.59 0.64 50.31 0.93 0.85 1.14 Kind in trouble 317 2.01 1.13-0.21 0.39 0.44 50.33 1.14 1.26 0.79 Hard time sleeping 319 1.98 1.29-0.24 0.62 0.62 50.66 1.06 0.99 1.11 Waiting your turn 321 1.92 1.20 0.28 0.54 0.56 51.31 1.04 0.95 1.05 Item-Total Std CFA Loadings Measure Infit Outfit Discrimination 40

Throw when mad 321 1.85 1.10 0.36 0.50 0.52 52.03 0.98 0.93 1.04 Hard to have fun 318 1.84 1.17 0.42 0.47 0.50 52.21 1.13 1.07 1.04 Afraid of laugh 322 1.76 1.13 0.49 0.54 0.60 53.18 1.11 1.09 1.04 Worthless 320 1.75 1.13 1.03 0.60 0.62 53.30 1.01 0.87 1.11 Scared to ask 315 1.74 1.20 1.05 0.47 0.47 53.42 1.41 1.25 0.99 No friends 317 1.72 1.20 1.07 0.48 0.53 53.68 1.27 1.09 1.10 Bully 320 1.59 0.99 2.33 0.44 0.47 55.48 1.14 1.02 1.02 Hurting self 313 1.39 0.90 5.96 0.44 0.43 58.98 1.20 1.00 1.09 Drugs 319 1.30 0.84 8.69 0.22 0.27 61.14 1.71 1.77 0.92 Alcohol 322 1.27 0.71 8.18 0.31 0.33 62.11 1.44 2.19 0.92 Note: Items listed in ascending order by item difficulty (Measure) Table 4.10 Comprehensive Item Analysis for the SFSS-Adult Caregiver Item N Mean St Dev Kurtosis Item-Total CFA Loadings Measure Infit Outfit Discrimination Disobey adults 265 3.00 1.27-0.89 0.76 0.81 42.53 0.75 0.74 1.32 Hard time c temper 263 2.94 1.28-0.90 0.74 0.79 42.60 0.74 0.72 1.35 Interrupt others 264 2.97 1.29-0.93 0.75 0.80 42.77 0.78 0.78 1.29 Hard time paying attention 263 2.84 1.31-1.02 0.74 0.76 43.58 0.82 0.81 1.28 Argue with adults 265 2.86 1.39-1.17 0.72 0.78 43.70 0.86 0.85 1.30 Feel unhappy or sad 267 2.79 0.98-0.03 0.62 0.62 44.24 0.62 0.70 1.31 Getting along w/ family 263 2.76 1.19-0.74 0.78 0.82 44.74 0.56 0.56 1.48 Get into trouble 267 2.72 1.20-0.73 0.68 0.74 44.75 0.82 0.82 1.13 Worry a lot 264 2.83 1.12-0.46 0.58 0.58 44.84 0.83 0.91 1.08 Lie to get things 264 2.71 1.31-0.99 0.70 0.76 45.35 0.98 1.00 1.02 41

Annoy other people 265 2.75 1.37-1.14 0.70 0.74 45.39 0.93 0.92 1.14 Hard time sitting still 259 2.72 1.36-1.17 0.62 0.66 46.37 1.17 1.16 0.82 Fights 265 2.63 1.32-1.06 0.74 0.79 46.54 0.85 0.84 1.25 Lose things you need 264 2.62 1.31-0.90 0.61 0.61 46.98 1.01 1.01 0.96 Eat a lot more or less 266 2.43 1.24-0.81 0.51 0.50 48.42 1.31 1.53 0.61 Hard time waiting turn 263 2.45 1.29-0.75 0.66 0.69 48.88 0.89 0.90 1.15 Feel tense 264 2.44 1.10-0.45 0.63 0.60 49.04 0.76 0.81 1.20 Have little or no energy 264 2.29 1.17-0.72 0.50 0.47 50.24 1.27 1.34 0.54 Afraid others would laugh 261 2.22 1.21-0.55 0.63 0.63 50.79 1.04 1.00 1.05 Threaten or bully others 261 2.13 1.17-0.57 0.61 0.70 51.84 0.97 0.90 1.05 Hang with kids in trouble 264 2.09 1.25-0.25 0.45 0.46 52.02 1.60 1.57 0.45 Feel nervous around people 263 2.16 1.10-0.36 0.42 0.43 52.18 1.18 1.34 0.68 Sleep a lot more 264 2.14 1.16-0.43 0.51 0.48 52.22 1.30 1.42 0.56 Throw things when mad 265 2.03 1.18-0.12 0.61 0.66 52.38 1.08 1.08 1.01 Don't have any friends 265 2.08 1.19-0.09 0.54 0.56 52.71 1.08 1.04 0.93 Hard time having fun 262 2.08 1.09-0.27 0.56 0.55 52.85 0.87 0.87 1.09 Cry easily 265 2.05 1.17 0.08 0.49 0.52 52.87 1.19 1.19 0.83 Feel worthless 260 1.99 1.07-0.74 0.61 0.56 53.56 0.93 0.90 1.09 Hard time sleeping 263 1.99 1.16-0.23 0.54 0.49 53.66 1.15 1.06 0.91 Too scared to ask in class 259 1.93 1.13 0.01 0.48 0.50 54.81 1.25 1.25 0.82 Think about hurting yourself 260 1.52 0.86 2.70 0.48 0.45 60.74 1.13 1.10 0.98 Use drugs non-medical 261 1.33 0.81 7.61 0.29 0.29 67.88 2.02 2.04 0.79 Drink alcohol 261 1.26 0.67 9.64 0.17 0.14 68.56 1.78 2.01 0.80 Note: Items listed in ascending order by item difficulty (Measure). 42

Table 4.11 Comprehensive Item Analysis for the SFSS-Clinician Item N Mean St Dev Kurtosis Item-Total CFA Loadings Measure Infit Outfit Discrimination Argue with adults 651 3.11 1.10-0.49 0.67 0.75 39.83 0.75 0.75 1.31 Disobey adults 648 3.10 1.02-0.20 0.64 0.73 40.03 0.67 0.68 1.39 Getting along w/ family 654 3.06 1.02-0.27 0.63 0.70 40.50 0.69 0.70 1.33 Feel unhappy or sad 649 3.01 0.96 0.05 0.54 0.53 41.10 0.76 0.78 1.26 Hard time c temper 650 2.96 1.16-0.68 0.64 0.72 41.69 0.89 0.88 1.13 Worry a lot 651 2.93 0.99-0.14 0.48 0.44 42.04 0.90 0.98 1.02 Get into trouble 651 2.91 1.06-0.37 0.61 0.69 42.29 0.78 0.79 1.29 Fights 651 2.79 1.12-0.58 0.62 0.69 43.73 0.87 0.87 1.13 Hard time paying attention 651 2.76 1.15-0.74 0.60 0.64 43.99 0.92 0.91 1.10 Interrupt others 651 2.76 1.13-0.67 0.59 0.66 44.07 0.92 0.91 1.03 Annoy other people 653 2.56 1.17-0.88 0.58 0.65 46.47 1.03 1.02 0.98 Lie to get things 651 2.55 1.18-0.67 0.57 0.63 46.66 1.07 1.04 0.92 Feel tense 651 2.43 1.01-0.54 0.55 0.49 48.09 0.83 0.86 1.22 Hang with kids in trouble 652 2.37 1.16-0.82 0.41 0.44 48.76 1.35 1.35 0.60 Hard time sitting still 653 2.35 1.19-0.83 0.54 0.58 49.11 1.18 1.20 0.83 Hard time waiting turn 652 2.27 1.12-0.73 0.56 0.61 50.01 1.04 1.01 1.00 Have little or no energy 648 2.27 1.05-0.40 0.40 0.35 50.12 1.18 1.20 0.71 Feel nervous around people 650 2.25 1.00-0.58 0.49 0.44 50.33 0.93 1.01 1.00 Afraid others would laugh 649 2.23 1.09-0.67 0.54 0.49 50.64 1.04 1.04 1.00 Hard time having fun 650 2.10 0.95-0.57 0.49 0.45 52.32 0.90 0.92 1.01 Feel worthless 651 2.10 1.03-0.56 0.58 0.54 52.37 0.89 0.86 1.12 Lose things you need 652 2.08 1.06-0.45 0.55 0.54 52.55 1.01 0.99 1.03 Threaten or bully others 653 2.07 1.06-0.24 0.54 0.60 52.72 1.03 0.97 0.98 43

Hard time sleeping 651 2.00 1.03 0.28 0.51 0.43 53.79 1.05 1.04 0.95 Don't have any friends 651 2.00 1.11 0.28 0.53 0.48 53.79 1.17 1.22 0.82 Sleep a lot more 651 1.98 1.03-0.36 0.42 0.38 54.04 1.20 1.26 0.78 Eat a lot more or less 650 1.96 1.06-0.09 0.49 0.47 54.20 1.16 1.22 0.88 Cry easily 652 1.94 1.07 0.39 0.46 0.42 54.53 1.25 1.25 0.76 Throw things when mad 651 1.94 1.05-0.14 0.54 0.57 54.63 1.08 1.04 0.99 Too scared to ask in class 649 1.80 0.99 0.67 0.48 0.43 56.66 1.13 1.14 0.93 Think about hurting yourself 651 1.45 0.79 3.34 0.42 0.39 63.45 1.17 1.03 0.99 Use drugs non-medical 650 1.33 0.79 7.25 0.21 0.20 66.82 1.84 1.85 0.77 Drink alcohol 653 1.27 0.72 9.99 0.24 0.21 68.67 1.73 1.83 0.86 Note: Items listed in ascending order by item difficulty (Measure). 44

Standard Errors of Measurement For the SFSS, the standard errors of measurement (SEMs) are presented in Table 4.12. For example, on the adult caregiver SFSS-33, we can say that the true score is between approximately ± 2 SEMs, or 2.62 points on a one to five point scale with 95% certainty. It is not surprising that the SEMs for the short forms are higher given their somewhat lower internal consistency reliabilities. The higher standard error for the youth versions also suggest that there is less certainty in the exact level of the youth s rating of severity at any given measurement instance. Having multiple measurement instances from the same youth allows direct comparisons to the youth s prior ratings. Table 4.12 Standard Errors of Measurement by SFSS Form and Informant SFSS Form Adult Caregiver Clinician Youth SFSS-33 2.62 2.34 3.29 SFSS Form A 3.58 3.28 4.52 SFSS Form B 3.59 3.03 4.57 Reliable Change Index The reliable change threshold for each form and informant on the SFSS is reported in Table 4.13. For example, the reliable change index (RCI) for the youth self-report score on the SFSS-33 is 5.36 points, giving us 75% confidence that a difference of more than 5.36 points is not due to chance. If the change is in a negative direction (i.e., decrease in score value) it represents an improvement in perceived youth symptoms and functioning, while a change in the positive direction (i.e., increase in score value) indicates that the youth s global severity is worsening. Table 4.13 Reliable Change Thresholds by SFSS Form and Informant SFSS Form Adult Caregiver Clinician Youth 75% 75% 75% SFSS-33 4.26 3.80 5.36 SFSS Form A 5.82 5.34 7.35 SFSS Form B 5.84 4.92 7.43 Test-Retest Reliability Another test of a scale s reliability is its test-retest correlation. For this purpose data are typically collected from the same group of individuals at two different time points. For scales that measure a more or less constant construct, such as an intelligence test, one would expect high correlations between the first time an individual was assessed and the second time, even if the time between those two measurement instances is substantial (e.g., a year). For a construct like symptom severity, change is expected over time, especially if the youth is in treatment. On average, one would not expect major changes in a short time period (e.g., two weeks). Since the average time interval of seven days 45

between administrations in our test-retest sample (SD = 3.8; range 1-14 days), we would expect a relatively high correlation of 0.7 or higher. Table 4.14 provides the correlations for our test-retest samples, in which correlations range between 0.68 and 0.87. These results suggest some variability over time, as expected since the youth are in treatment, as well as a relatively strong relationship between the two assessments. For example, approximately 75% of the variance in adult caregiver ratings at the second assessment can be explained by using the ratings from the first assessment. In comparison, the testretest reliability is the strongest for adult caregivers, followed by clinicians, and the lowest for youth. It is important to note that the observed correlations with samples of 54-88 participants are only approximate due to sampling error. Future analyses with larger samples will have to confirm the robustness of these results. Table 4.14 SFSS Test-Retest Correlations Respondent r N Adult Caregiver 0.87 54 Clinician 0.74 88 Youth 0.68 84 Evidence of Validity The objective of this section is to provide evidence for the construct validity of the SFSS. The construct the SFSS is intended to measure is complex. Weiss, Susser & Catron s (1998) model of psychopathology suggests that it has wide-band common features as well as narrow-band specific features (internalizing-externalizing and more specific aspects such as anxiety versus depression). According to the DSM-IV-TR, psychopathology occurs in distinct subtypes, each defined by distinct profiles of symptoms and deficits. The psychometric sample includes youth with a wide variety of problems and the SFSS is a broad spectrum estimate, including anxiety, depression, behavior problems, as well as other problems youth typically experience often at the same time. Thus, we did not expect to find as clear of a factor structure as found in specialized measures, such as Brekke s (2002) five aspects of social functioning in schizophrenics, for example. Interpretation of the factor analyses presented below should take this complexity into account. Scree Plot A combined scree plot of eigenvalues for the three informants of the SFSS, youth, adult caregiver and clinician, appears in Figure 4.1. The plots suggests that the three respondents respond in a similar fashion, with one large and robust principle component, and four additional ones that contribute to less than 4% of the variance. Since we used theoretical considerations prior to the empirical analyses when we determined whether 46

each of the items should be considered part of either the internalizing or externalizing subscale, we present the results of the CFA next. Figure 4.1 Combined Scree Plot of Eigenvalues for SFSS Confirmatory Factor Analysis We evaluated three different models using a confirmatory factor analysis (CFA) conducted with SAS CALIS in a mode emulating Bentler & Wu s (1995) EQS. The first model was a one-factor model where all 33 items were loading onto the one single factor. The second model is also based on one factor but only those items that were not ambiguous in regard to the internalizing or externalizing subscales were included (see Table 4.1). The third model included the same items as the second model but is based on two correlated factors (internalizing and externalizing) according to the theory (see Table 4.1). The second and the third model are nested and can be compared directly, while the first model can only indirectly compared to the other two. As can be seen in Tables 4.15, 4.16, and 4.17, none of the models meet the established criteria of fit (CFI, GFI, and RMSEA) in a satisfactory way. It is interesting to note that in the youth sample, the factor structure is the clearest, while in the clinician sample the models fit the data the poorest. Comparatively, however, the third model has a clear superior fit relative to the other two models, which, for the last two models is confirmed 47

by the significant chi-square differences tests. We consider this indirect evidence and support for the validity and use of the two subscale scores. Given this and the high internal reliability of the subscales and the difference of their average scores for all three respondent types, we feel comfortable recommending the use of the two subscales as a general orientation in addition to the total score. The standardized factor loadings for the items on the internalizing and externalizing factors in the third model ranged for the adult caregiver version from 0.47 to 0.76 for internalizing, and 0.43 to 0.83 for externalizing. For the clinician version, they ranged from 0.52 to 0.74 and 0.47 to 0.81 respectively. The range for the youth version is 0.44 to 0.73 and 0.52 to 0.72. These loadings should be considered with caution, especially for the adult caregiver and clinician version, because of the unsatisfactory fit of the overall model to the data. Table 4.15 Evaluation of the SFSS Factor Structure Adult Caregiver Model χ 2 df χ 2 / χ 2 Bentler Joreskog diff df CFI GFI RMSEA One-Factor All 33 Items 2716.0 495 0.72 0.62 0.10 One-Factor Selected Items 2290.7 405 5.66 0.75 0.63 0.10 Two Correlated Factors 1492.4 404 3.69 798.3 0.86 0.80 0.08 For the CFI and GFI, values greater than 0.90 indicate good fit between a model and the data. For the RMSEA, a value of 0.05 indicates close fit, 0.08 fair fit, and 0.10 marginal fit (Browne & Cudeck, 1993). Table 4.16 Evaluation of the SFSS Factor Structure Clinician Model χ 2 df χ 2 / χ 2 Bentler Joreskog diff df CFI GFI RMSEA One-Factor All 33 Items 5217.0 495 0.55 0.51 0.13 One-Factor Selected Items 4294.7 405 10.60 0.59 0.54 0.13 Two Correlated Factors 2629.1 404 6.51 1665.6 0.76 0.73 0.10 For the CFI and GFI, values greater than 0.90 indicate good fit between a model and the data. For the RMSEA, a value of 0.05 indicates close fit, 0.08 fair fit, and 0.10 marginal fit (Browne & Cudeck, 1993). 48

Table 4.17 Evaluation of the SFSS Factor Structure Youth Model χ 2 df χ 2 / χ 2 Bentler Joreskog diff df CFI GFI RMSEA One-Factor All 33 Items 2369.4 495 0.71 0.68 0.09 One-Factor Selected Items 1957.8 405 4.83 0.75 0.70 0.09 Two Correlated Factors 1291.2 404 2.88 796.3 0.86 0.83 0.07 For the CFI and GFI, values greater than 0.90 indicate good fit between a model and the data. For the RMSEA, a value of 0.05 indicates close fit, 0.08 fair fit, and 0.10 marginal fit (Browne & Cudeck, 1993). Convergent and Discriminant Validity In order to provide additional evidence for the construct validity of the SFSS, we also evaluated criteria for discriminant and convergent validity. Discriminant validity showed that using the scale the construct can be measured in a way that it can be differentiated from related but distinct constructs. In convergent validity, we examined the degree to which the operationalization is similar to (converges on) other operationalizations to which it should be theoretically similar. Evidence for convergent validity was established by showing that the scale scores correlated highly with other established scales that measure the same or a very similar construct. Sometimes, this is also referred to as concurrent validity. Evidence for this convergent and discriminant validity for the SFSS is presented in Chapter 3. Using the SFSS Scoring Use the self-scoring forms (Tables 4.19, 4.20, and 4.21) to calculate the SFSS Total Score. These tables are also available with the measures in Appendix B. Enter the value for the selected responses in fields A-GG for the SFSS-33, or fields A-Q or A-P for Form A or Form B respectively. Calculate the total score by adding the scores and then divide the sum by the number of items, 33 for the SFSS-33, 17 for the Form A, and 16 for Form B. Finally, do a linear transformation by multiplying the average score by the factor U and then add the factor V as shown in Table 4.18. There are no reverse coded items in the SFSS. The self-scoring forms are also available in Appendix B: Measures and Self- Scoring Forms. Table 4.18 Factors for Linear Transformation of SFSS Scores U V Youth 18.7285 13.369 Adult Caregiver / Clinician 15.6691 26.493 49

Use the scoring forms in the case where measures are fully completed (100% response rate). Otherwise, in cases with missing data, scoring can be done by computing the mean of completed items. Determining when too much missing data occurs for computing summary scores is at the discretion of the user. The analyses presented in this chapter required 85% of the items to have valid answers. Table 4.19 Self-Scoring Form SFSS-33 Item Never Hardly Ever Values for Responses Sometimes Often Very Often Enter value for selected responses here and calculate scores as instructed Subscale E Subscale I 1 1 2 3 4 5 A 2 1 2 3 4 5 B 3 1 2 3 4 5 C 4 1 2 3 4 5 D 5 1 2 3 4 5 E 6 1 2 3 4 5 F 7 1 2 3 4 5 G 8 1 2 3 4 5 H 9 1 2 3 4 5 I 10 1 2 3 4 5 J 11 1 2 3 4 5 K 12 1 2 3 4 5 L 13 1 2 3 4 5 M 14 1 2 3 4 5 N 15 1 2 3 4 5 O 16 1 2 3 4 5 P 17 1 2 3 4 5 Q 18 1 2 3 4 5 R 19 1 2 3 4 5 S 20 1 2 3 4 5 T 21 1 2 3 4 5 U 22 1 2 3 4 5 V 23 1 2 3 4 5 W 24 1 2 3 4 5 X 25 1 2 3 4 5 Y 26 1 2 3 4 5 Z 27 1 2 3 4 5 AA 28 1 2 3 4 5 BB 29 1 2 3 4 5 CC 30 1 2 3 4 5 DD 31 1 2 3 4 5 EE 32 1 2 3 4 5 FF 33 1 2 3 4 5 GG 50

Subscale E Subscale I Total Score Sum of A, D, F-I, L, M, R, T, V-Y, CC, and DD HH / 16 Perform calculation noted below HH Sum of B, C, E, K, N, P, Q, S, U, Z-BB, FF, and GG For Youth, multiply by 18.7285 and add 13.369 For Adult Caregiver or Clinician, multiply by 15.6691 and add 26.493 SFSS Subscale E Score = JJ SFSS Subscale I Score = MM SFSS-33 Total Score = QQ II JJ KK / 14 KK Perform calculation noted below MM Sum of J, O, and EE LL Sum of HH, KK, NN OO / 33 Perform calculation noted below NN OO PP QQ Table 4.20 Self-Scoring Table - SFSS Form A Values for Responses Item Never Hardly Ever Sometimes Often Very Often Enter value for selected responses here and calculate scores as instructed 1 1 2 3 4 5 A 2 1 2 3 4 5 B 3 1 2 3 4 5 C 4 1 2 3 4 5 D 5 1 2 3 4 5 E 6 1 2 3 4 5 F 7 1 2 3 4 5 G 8 1 2 3 4 5 H 9 1 2 3 4 5 I 10 1 2 3 4 5 J 11 1 2 3 4 5 K 12 1 2 3 4 5 L 13 1 2 3 4 5 M 14 1 2 3 4 5 N 15 1 2 3 4 5 O 16 1 2 3 4 5 P 51

17 1 2 3 4 5 Q Sum of A-Q: R R / 17: For Youth, multiply S by 18.7285 and add 13.369 For Adult Caregiver or Clinician, multiply S by 15.6691 and add 26.493 SFSS Form A Total Score = T S T Table 4.21 Self-Scoring Form- SFSS Form B Values for Responses Item Never Hardly Ever Sometimes Often Very Often Enter value for selected responses here and calculate scores as instructed 1 1 2 3 4 5 A 2 1 2 3 4 5 B 3 1 2 3 4 5 C 4 1 2 3 4 5 D 5 1 2 3 4 5 E 6 1 2 3 4 5 F 7 1 2 3 4 5 G 8 1 2 3 4 5 H 9 1 2 3 4 5 I 10 1 2 3 4 5 J 11 1 2 3 4 5 K 12 1 2 3 4 5 L 13 1 2 3 4 5 M 14 1 2 3 4 5 N 15 1 2 3 4 5 O 16 1 2 3 4 5 P Sum of A-P: Q Q / 16: For Youth, multiply R by 18.7285 and add 13.369 For Adult Caregiver or Clinician, multiply R by 15.6691 and add 26.493 SFSS Form B Total Score = S R S 52

Interpretation The SFSS assesses symptoms and functioning during the previous two weeks. Scores on the SFSS can fluctuate significantly from one administration to the next. Thus, it is important to interpret whether these changes represent clinically significant change and if so, what may have caused it. It will also be important to administer the SFSS frequently so that general trends can be reliably assessed. A negative trend indicates that the youth is experiencing fewer problem behaviors and emotions. Low, Medium, High Scores In general, the scores of the SFSS can range from 42 to 105 for adult caregivers and clinicians, and 32 to 107 for youths, where a high score represents high severity while a low score indicates low severity. The low, medium, high scores presented in Table 4.22 apply to all forms of the SFSS, and help to judge whether a score should be considered relatively low, medium, or high, compared to the current psychometric sample. Respondents who rate their (or the youth s) severity as high are experiencing behaviors or emotions that are causing problems in their life. When a youth reports low severity it indicates that the youth is experiencing few problem behaviors or emotions. In addition, the scores can be interpreted relative to the CBCL. The high correlation with the CBCL and linear transformation conducted for the total SFSS scores allow the use of the criteria established for the CBCL and YSR. In general, scores of 65 or higher are considered to be in the clinical range. Table 4.22 SFSS Low, Medium, and High Scores Scale / Subscale Low Medium High SFSS-Youth < 45 45-63 > 63 Internalizing < 41 41-64 > 64 Externalizing < 46 46-68 > 68 SFSS- Adult Caregiver < 58 58-73 > 73 Internalizing < 54 54-70 > 70 Externalizing < 60 60-80 > 80 SFSS-Clinician < 57 57-69 > 69 Internalizing < 53 53-68 > 68 Externalizing < 59 59-75 > 75 Percentile Ranks Percentile ranks for the youth, adult caregiver and clinician psychometric samples appear in Table 4.23. For example, as shown in the table for youth respondents, a score of 50 is in the 38 th percentile. This means that in the psychometric sample, 38% scored 50 or lower and 62% scored higher. 53

Table 4.23 SFSS Percentile Ranks for Total Scores SFSS-Youth SFSS-Adult Caregiver SFSS-Clinician Score Percentile Score Percentile Score Percentile 32 1 43 1 43 1 33 2 44 2 44 1 34 3 45 3 45 1 35 5 46 4 46 2 37 8 47 5 47 2 38 10 48 6 48 3 39 12 49 7 49 5 40 13 50 8 50 7 41 15 51 10 51 9 42 18 52 11 52 12 43 21 53 13 53 14 44 23 54 15 54 17 45 26 55 18 55 19 46 29 56 20 56 22 47 31 57 22 57 27 48 32 58 25 58 32 49 35 59 28 59 36 50 38 60 31 60 39 51 42 61 34 61 43 52 45 62 38 62 48 53 47 63 42 63 52 54 50 64 45 64 56 55 53 65 49 65 60 56 56 66 52 66 63 57 59 67 56 67 67 58 62 68 61 68 71 59 65 69 64 69 75 60 67 70 68 70 78 61 69 71 70 71 81 62 72 72 72 72 84 63 74 73 75 73 88 64 76 74 78 74 89 65 78 75 81 75 91 66 79 76 82 76 92 67 81 77 84 77 94 68 84 78 86 78 95 69 85 79 88 79 96 70 87 80 89 80 96 71 88 81 91 81 97 72 89 82 92 82 98 73 90 83 93 84 98 74 91 84 94 85 99 75 92 85 95 86 99 76 94 86 96 88 99 77 94 87 96 90 99 78 95 88 97 91 99 79 95 89 97 80 95 90 98 81 96 91 98 83 96 92 99 84 97 94 99 85 98 96 99 86 98 103 99 87 98 88 99 91 99 92 99 101 99 Note: Percentiles on the SFSS short forms are extremely similar. 54

References Achenbach, T. M. (1991). Integrative guide for the 1991 CBCL/4-18, YSR, and TRF profiles. Burlington, VT: University of Vermont, Department of Psychiatry. Achenbach, T. M., McConoughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213-232. American Psychological Association (2000). Diagnostic and statistical manual of mental disorders test revision DSM-IV-TR (text revision). Washington, DC: Author. Bentler, P. M. & Wu, E. J. C. (1995). EQS for Windows user s guide. Encino, CA: Multivariate Software, Inc. Brekke, J. S., Long, J. D., & Kay, D. (2002). The structure and invariance of a model of social functioning in schizophrenia. Journal of Nervous and Mental Disease, 190(2), 63-72. Browne, M. W., & Cudeck, R. (1993). Alternative ways of accessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park: Sage. Doucette, A., & Bickman, L. (2001). Child Adolescent Measurement System (CAMS). Nashville, TN: Author. Meyer, G. J., Finn, S. E., Eyde, L., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, E. J., Kubiszyn, T. W., & Reed, G. M. (2001). Psychological testing and psychological assessment: A review of evidence and issues. American Psychologist, 56, 128-165. Seligman, M. E. P. (2002). Using the New Positive Psychology to Realize Your Potential for Lasting Fulfillment. New York, NY: Simon & Schuster. Weiss, B., Susser, K. & Catron, T. (1998). Common and specific features of childhood psychopathology. Journal of Abnormal Psychology, 107(1), 118-127. 55