Chapter 9. Youth Counseling Impact Scale (YCIS)

Chapter 9 Youth Counseling Impact Scale (YCIS) Background Purpose The Youth Counseling Impact Scale (YCIS) is a measure of perceived effectiveness of a specific counseling session. In general, measures of impact are concerned with clients internal reactions to sessions which, logically, must intervene between in-session events and the long-term effects of treatment (Stiles et al., 1994, p.175). The YCIS specifically asks the youth to report on the positive impact he or she thinks the counseling sessions have had in terms of (1) insight into problems and possible solutions, and (2) behavioral, cognitive and emotional changes made following the previous session. The client s perception that treatment is working is an important factor in the treatment process. Theory The most commonly used session impact measures are the Session Evaluation Questionnaire (SEQ; Stiles, 1980) and the Session Impact Scale (SIS; Elliot & Wexler, 1994). The SEQ consists of several bipolar adjective pairs that describe perceptions of session impact on two dimensions: depth and smoothness (Stiles, 1980). The SIS assesses perceptions of session impact on the dimensions of hindering versus helpful impacts (Elliot & Wexler). The latter has two sub-dimensions: task impacts and relationship impacts. Unlike the SEQ which focuses on general emotional reactions to sessions, the SIS targets the specific content of therapy sessions (Elliot & Wexler). Both of these measures, however, were designed for adults. Measures of session impact for youth are seriously lacking. To our knowledge, the only impact measure that has been developed specifically for adolescents is the Session Evaluation Form (SEF) by Bussell and Kurtines, which was adapted from the SIS (see Bussell, 2000). The limitation of the SEF is that it was developed specifically for group sessions. Additionally, like most session impact scales the SEF only assesses the immediate reaction after a session but does not examine the effects of changes that occur in the weeks following a session. These changes, however, are especially relevant in the mental health treatment of youth. The YCIS was developed for use in individual counseling sessions with youth ages 11-18 years and includes an assessment of changes that occur both immediately after and in the weeks following a session. For our purposes, we defined helpful counseling impact as the perceived immediate effect of a counseling session on client insight (understanding of feelings, relationships, and problems) and client change (positive behavioral, cognitive, and emotional changes 109

that occur in the weeks immediately following the counseling session). The former was derived from the task impact dimension described by Elliot and Wexler (1994). History of Development The development of the YCIS began with a review of the relevant literature on session impact. Based on this review and the intended purpose of this measure for the overall measurement battery, 12 potential items were identified. After an internal review by our research team and an external review by the clinical advisory board of our partner organization, the list was reduced to ten items. These ten items were evaluated as part of the psychometric study (see Chapter 2). All ten items demonstrated good psychometric properties and the theoretical construct was confirmed by the data as described further below. Based on the recent increase in the use of a strength-based approach, we added an additional item that asks specifically about insight into the youth s strengths. Since this item was added after completion of the current psychometric study, we are not able to report on its psychometric properties in regard to this 11 item scale at this time. Structure The YCIS measures perceived helpful counseling impact using ten youth appropriate items, five of which assess insight immediately after a session and five that measure emotional, cognitive, and behavioral changes in the two weeks prior to the current session. Each item is rated on a five-point Likert-type scale, with higher scores corresponding to greater positive impact. Responses range from one (Not at All a Problem) to five (Totally). While not part of the psychometric evaluation, we recommend the addition of the 11 th item, which specifically asks about insight into strengths ( I know have a better understanding of my strengths ). Factor analysis with the test sample has confirmed that the YCIS assesses a total score of counseling impact and two related but distinct subscales, insight and change. The YCIS Total Score is calculated as the mean of all 11 items. The two subscale scores are both means of item subsets: Insight (items one to six) and Change (items 7 to 11). The psychometrics described are based on the complete sample of the psychometric study with the 10-item version, and required 85% of the items to have valid answers. See Chapter 2 for more detail on the psychometric sample and test development procedures. Administration The YCIS is a written self-report treatment phase measure to be administered at least two weeks after treatment starts. As can be seen in Table 9.1, no intake, discharge, or followup version exists. The YCIS is completed by the youth only. The suggested frequency of administration is every other week or at least once a month. 110

Table 9.1 Administration of YCIS by Phase Intake Treatment Discharge Follow-Up Y A C Y A C Y A C Y A C Y = Youth (age 11-18); A = Adult Caregiver; C = Clinician Recommended Frequency: Every two weeks or at least once a month The suggested administration schedule of all the measures in the Peabody Treatment Progress Battery is presented in Appendix A. All PTPB measures with self-scoring tables can be found in Appendix B: Measures and Self-Scoring Forms. Description Basic Descriptives As seen in Table 9.2, the mean of the YCIS Total Score as well as the scores for each subscale are about 0.5 units above the scale middle of 3.0 (3.57, 3.56, and 3.47 respectively). However, the overall distribution is close to a normal distribution with no worrisome skewness or kurtosis and scores across the full range of possible scale scores. The correlation between the two subscales is relatively large (r = 0.72, N = 446; p < 0.001). However, the confirmatory factor analysis (CFA) reported below provides evidence for two distinct sub-factors that together compose the general counseling impact factor. Table 9.2 Descriptive Statistics for YCIS Summary Scores N Mean Std Dev Skewness Kurtosis Min Max YCIS Total Score 499 3.57 1.00-0.47-0.38 1 5 Insight Subscale Score 499 3.66 1.08-0.65-0.26 1 5 Change Subscale Score 457 3.47 1.08-0.40-0.51 1 5 Note: Data represented in this table reflect the 10-item version of the YCIS. Quartiles Quartiles appear in Table 9.3. High scores are those in the top quarter, with low scores in the bottom quarter. For the YCIS Total Score, a score greater than 4.4 is considered high, whereas a score less than 3.0 is considered low. To aid interpretation, the quartiles were used to create low, medium, and high scores and percentile ranks based on comparison to the psychometric sample. This information is presented in the last section of this chapter. 111

Table 9.3 YCIS Quartiles Quartile YCIS Insight Change Total Score Subscale Subscale 100 % Max 5.0 5.0 5.0 75 % Q3 4.4 4.6 4.4 50 % Median 3.7 3.8 3.6 25 % Q1 3.0 3.0 2.8 00 % Min 1.0 1.0 1.0 Note: Data represented in this table reflect the 10-item version of the YCIS. Evidence of Reliability Reliability Coefficients The Cronbach s alpha internal consistency reliability correlations are presented in Table 9.4. These alphas suggest a satisfactory degree of internal consistency for the total score and the subscales. Table 9.4 Cronbach s Alphas for the YCIS Unstandardized Standardized Scale / Subscale Alpha Alpha YCIS 0.92 0.92 YCIS Insight 0.91 0.91 YCIS Change 0.86 0.87 Note: Data represented in this table reflect the 10-item version of the YCIS. Comprehensive Item Psychometrics Table 9.5 presents the comprehensive item psychometrics. Shaded cells indicate that a criterion was out of the range of sought values, as described previously in Table 2.2 in Chapter 2. Only items with two or more shaded cells are considered problematic. Almost all items in the YCIS showed satisfactory scale characteristics. The only exception is item number 10 which asks about changes in school (infit and outfit criteria above 1.5). This is not surprising since the data were collected during the summer when school was not in session. Thus, it is likely that respondents were not clear how to respond to this item. As a consequence, this item was revised to include other places outside of the client s home beside school. 112

Table 9.5 Comprehensive Item Analysis for the YCIS Item N Mean St Dev Kurtosis Item-Total Std CFA Loadings Measure Infit Outfit Discrimination Improved behavior at school 407 3.25 1.55-1.39 0.66 0.63 0.43 1.75 1.76 0.35 Used what I have learned 439 3.48 1.27-0.72 0.77 0.87 0.13 0.74 0.76 1.24 Understand about somebody else 438 3.50 1.29-0.67 0.72 0.79 0.11 0.98 1.06 1.01 Have felt better about myself 439 3.51 1.32-0.80 0.73 0.82 0.08 0.9 0.96 1.08 Learned about myself 440 3.58 1.32-0.69 0.71 0.78 0.01 1.02 1.14 0.99 Have tried things suggested 441 3.58 1.28-0.57 0.67 0.71-0.01 1.26 1.39 0.70 Improved behavior at home 439 3.59 1.23-0.54 0.70 0.76-0.04 1.01 1.13 0.91 Understand my feelings better 439 3.63 1.27-0.54 0.76 0.88-0.07 0.69 0.71 1.32 Understand my problems better 440 3.78 1.20-0.29 0.72 0.86-0.31 0.83 0.80 1.14 Better idea how to deal 441 3.80 1.22 0.02 0.71 0.82-0.34 0.89 0.90 1.16 Notes: Items listed in descending order by item difficulty (Measure). Data represented in this table reflect the 10-item version of YCIS 113

Standard Errors of Measurement For the YCIS Total Score, the standard error of measurement (SEM) is 0.28 points. With 95% confidence, we can say that that the true score is between approximately ±2 SEMS, or 0.55 points on a one to five point scale. The SEMs for the YCIS subscales are 0.32 points for Insight (95% confidence interval between approximately ±2 SEMS, or 0.63 points) and 0.40 points for Change (95% confidence interval between approximately ±2 SEMS, or 0.78 points). Reliable Change Index The reliable change threshold is 0.45 points with 75% confidence for the YCIS Total Score, and gives us 75% confidence that a difference of more than 0.45 points is not due to chance. For the subscale scores, the reliable change index (RCI) is 0.53 for Insight and 0.66 for Change. If the change is in a positive direction (i.e., increase in score value) it represents an improvement in perceived positive counseling impact while a change in the negative direction (i.e., reduction in score value) indicates that the level of perceived impact is declining. Test-Retest Reliability Not available at this time. Evidence of Validity Scree Plot The scree plot (Figure 9.1) clearly provides support for the general primary factor. The secondary factors are more subtle and are in general difficult to detect with this type of exploratory method. However, the pattern of the factor loadings of the items on the second factor with an eigenvalue of approximately one provide some evidence for the separation of the 10 items into two distinct factors. The first five items (Insight) all have negative loadings on the second factor while the last five items (Change) all have positive loadings. The final factor structure was tested using confirmatory factor analysis (CFA). 114

Eigenvalues 6.5 6 5.5 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Number Figure 9.1 Scree Plot of Eigenvalues for YCIS Confirmatory Factor Analysis The expected factor structure inferred from the theory was a hierarchical G-Factor model with the two secondary factors (insight and change) loading on a higher-order counseling impact factor. This structure was well supported by the data. Confirmatory factor analysis (CFA) was conducted with SAS CALIS in a mode emulating Bentler & Wu s (1995) EQS. We ran a one-factor model as well as the G-Factor model to evaluate how well each model could explain the observed data. For the G-Factor model the loadings of the lower-order factors on the general counseling impact factor were set to be equal for the model to be identified. The superiority of the fit indices of the G-Factor model clearly provides support for the respective factor structure. The G-model demonstrates an excellent fit for these data while the fit indices for the one-factor model are less satisfactory. The χ 2 difference test further provides evidence for the G-Factor model compared to a one-factor model (χ 2 diff = 266, df = 1, p <.001). The two models are nested because the G-Factor model is mathematically the same as a model with two correlated factors and the one-factor model is the same as a two-factor model with perfectly correlated factors (correlation constrained to be 1). Since the difference is statistically significant the less constrained G-Factor model is preferred. The factor loadings (standardized estimates) for each manifest variable can be found in Tables 9.6 and 9.7. All loadings were high (between 0.63 and 0.88). The loading for the Insight and Change Factors on the general Counseling Impact Factor was 0.67 6. 6 Note that the loadings of the subscales on the general factor have set to be equal because the model would otherwise not be identified and inestimable. The standardized estimates for the loadings provided by PROC CALIS are 0.85 for Insight and 0.94 for Change. 115

Table 9.6 Evaluation of the YCIS Factor Structure Scale χ 2 df χ 2 / χ 2 Bentler diff df CFI YCIS One-Factor Model YCIS G-Factor Model Joreskog GFI RMSEA 371.55 35 10.62 0.89 0.82 0.15 103.36 34 3.04 268.19 0.98 0.96 0.07 For the CFI and GFI, values greater than 0.90 indicate good fit between a model and the data. For the RMSEA, a value of 0.05 indicates close fit, 0.08 fair fit, and 0.10 marginal fit (Browne & Cudeck, 1993). Data represented in this table reflect the 10-item version of the YCIS. Table 9.7 Loadings for the G-Factor Confirmatory Model of YCIS Insight Change Impact Item/Subscale Unstandar dized Standardiz ed 1 Learned about self 1.00 (--) 0.78 2 Understand other 1.01 (.06) 0.79 3 Understand feelings 1.13 (.05) 0.89 4 Understand problems 1.10 (.05) 0.85 5 How to deal 1.06 (.05) 0.83 Unstandar dized Standar dized 6 Tried things 1.00 (--) 0.71 Unstandar dized Standardiz ed 7 Felt better about self 1.16 (.07) 0.82 8 Used what learned 1.23 (.07) 0.87 9 Improved at home 1.07 (.07) 0.76 10 Improved in school 0.90 (.07) 0.64 Insight 0.67 (.03) 0.86 Change 0.67 (.03) 0.94 Note: Data represented in this table reflect the 10-item version of the YCIS. 116

Scoring the YCIS Scoring Use Table 9.8 to calculate the YICS Total Score as well as the two subscale scores. Enter the value for the answer choices in fields A-F and I M and calculate fields G, H, and N- Q as instructed. There are no reverse coded items in the YCIS. The self-scoring forms are also available in Appendix B: Measures and Self-Scoring Forms. Use the scoring form in the case where measures are fully completed (100% response rate). Otherwise, in cases with missing data, scoring can be done by computing the mean of completed items. Determining when too much missing data occurs for computing summary scores is at the discretion of the user. The analyses presented in this chapter required 85% of the items to have valid answers. Table 9.8 YCIS Self-Scoring Form Values for Responses Item Not at all Only a little Somewhat Quite a bit Totally Enter value for selected response here and calculate scores as instructed 01 1 2 3 4 5 A 02 1 2 3 4 5 B 03 1 2 3 4 5 C 04 1 2 3 4 5 D 05 1 2 3 4 5 E 06 1 2 3 4 5 F Sum of A-F: G G / 6: H 07 1 2 3 4 5 I 08 1 2 3 4 5 J 09 1 2 3 4 5 K 10 1 2 3 4 5 L 11 1 2 3 4 5 M Sum of I-M: N N / 5: O H + O: YCIS Total Score = Q YCIS-Insight Subscale Score = H YCIS-Change Subscale Score = O P / 2: P Q 117

Interpretation The YCIS is a session-based instrument and assesses the youth s perceptions of overall counseling impact, insight attained during the current session, and certain behavioral, cognitive and emotional changes during the previous two weeks. Thus, the YCIS can be used to evaluate specific sessions and be linked to the events in that session. This is especially true for the items that represent the Insight Subscale. This can be a useful tool for clinicians to determine if specific interventions or clinician behaviors in a session instill a feeling in clients that the treatment is working for them. The fact that the YCIS is session-based also means that the score can fluctuate significantly from one administration to the next. Thus, it is important to see if these changes represent reliable change (described above) and if so, what may have caused the change. It will also be important to administer the YCIS frequently so that general trends can be reliably assessed. A positive trend indicates that the youth is increasingly feeling that the sessions have a positive impact. The scores on the YCIS can range from one to five, where a five represents high positive impact while one indicates low impact. The total score and percentile ranks presented below (see Tables 9.9 and 9.10) help to judge whether a score should be considered relatively low, medium, or high. Youths who rate counseling impact as high believe that their treatment sessions have a positive impact on the way they feel about themselves, their insight into their problems and possible solutions, and on the way they behave. When youth report low impact, it does not necessarily mean that the treatment had no effect, but may indicate that youth had little or no understanding of the positive impact the treatment had on them. However, if youth believe the treatment has no positive impact, they may become less motivated to be in or even be resistant to treatment. Youth may also be less likely to make changes in behavior or the way they think and feel about themselves. Low, Medium, High Scores Based on the psychometric sample, a total score greater than 4.4 is considered high, which indicates that the youth reports a strong positive counseling impact. If the total score is less than 3.0, it is considered low, indicating that the youth s perception of counseling impact is lower than the psychometric study sample. All scores in between represent a medium level of perceived impact. Low, medium and high scores are listed in Table 9.9. Table 9.9 YCIS Low, Medium, and High Scores Scale/Subscales Low Medium High YCIS Total Score < 3.0 3.0-4.4 > 4.4 Insight < 3.0 3.0-4.6 > 4.6 Change < 2.8 2.8-4.4 > 4.4 Note: Data represented in this table reflect the 10-item version of the YCIS. 118

Percentile Ranks Percentile ranks appear in Table 9.10. For example, a score of 2.5 is in the 14 th percentile. This means that in the psychometric sample, 14% of youth scored 2.5 or lower and 86% scored higher. Table 9.10 YCIS Percentile Ranks for Total Scores Score Percentile Score Percentile Score Percentile Score Percentile 1.0 1 2.4 12 3.4 40 4.3 70 1.1 1 2.4 13 3.4 41 4.3 72 1.2 2 2.5 14 3.5 43 4.4 74 1.3 2 2.6 16 3.6 46 4.4 76 1.4 3 2.7 18 3.6 47 4.5 77 1.5 3 2.8 19 3.7 49 4.6 78 1.6 4 2.8 20 3.7 50 4.6 80 1.7 4 2.9 21 3.8 52 4.7 82 1.8 5 2.9 22 3.8 54 4.7 83 1.9 6 3.0 27 3.9 56 4.8 85 2.0 7 3.1 31 3.9 58 4.8 86 2.1 8 3.1 32 4.0 61 4.9 87 2.1 9 3.2 34 4.1 64 5.0 93 2.2 9 3.2 35 4.1 65 2.2 10 3.3 37 4.2 67 2.3 11 3.3 38 4.2 69 Note: Data represented in this table reflect the 10-item version of the YCIS References Bentler, P. M. & Wu, E. J. C. (1995). EQS for Windows user s guide. Encino, CA: Multivariate Software, Inc. Browne, M. W. & Cudeck, R. (1993). Alternative ways of accessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park: Sage. Bussell, J. R. (2000). Exploring the role of therapy process and outcome in interventions that target adolescent identity and intimacy. Dissertation Abstracts International: Section B: The Sciences and Engineering, 61(10-B), 5553. (UMI No. 9991552). Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. 119

Elliott, R., & Wexler, M. (1994). Measuring the impact of sessions in processexperiential therapy of depression: The Session Impacts Scale. Journal of Counseling Psychology, 41, 166-174. Stiles, W. B. (1980). Measurement of the impact of psychotherapy sessions. Journal of Consulting and Clinical Psychology, 48, 176-185. Stiles, W. B., Reynolds, S., Hardy, G. E., Rees, A., Barkham, M., & Shapiro, D. A. (1994). Evaluation and description of psychotherapy sessions by clients using the Session Evaluation Questionnaire and the Session Impacts Scale. Journal of Counseling Psychology, 41, 175-185. 120