An Evaluation of the Brief Service Program at Children s Centre Thunder Bay. Children s Centre Thunder Bay. Conducted by. Suzanne Chomycz, M.A.

Transcription

1 An Evaluation of the Brief Service Program at Children s Centre Thunder Bay Children s Centre Thunder Bay Conducted by Suzanne Chomycz, M.A. Lakehead University 1

2 Acknowledgements Children s Centre Thunder Bay (CCTB) would like to thank the Intake and Brief Services clinical staff for their contributions and data collection: Joni Nelson, Joe Houlton, Donna Strickland, Angela Racco, Jaymee Martin, Andy Meisner, and Kristin Rasmussen. Children s Centre Thunder Bay (CCTB) would like to thank the Ontario Centre of Excellence for Child and Youth Mental Health for their funding support. Thank you to Ilana Smyth and Marie- Josée Emard from the Ontario Centre of Excellence for Child and Youth Mental Health for their ongoing support and guidance throughout the evaluation. Finally, thank you to all the families who participated, and offered their time and contributions during this evaluation. 2

3 Table of Contents Introduction 4 Method 5 Results, Discussion, and Interpretation 7 Conclusions 12 References 13 Figure 1 15 Table

4 Introduction An Evaluation of the Brief Service Program at Children s Centre Thunder Bay CCTB is a community based agency in Thunder Bay, Ontario that offers a range of clinical services to children, adolescents, and their families from birth to age 17 who are experiencing emotional, behavioural, and/or developmental issues. The Brief Services program at CCTB has been in existence for over 10 years and was designed to provide more immediate and briefer interventions for clients with a wide range of less severe mental health problems. At CCTB, Brief Services are defined as a maximum of four treatment sessions within a two month period. The intervention models used by therapists within this program may include, but are not limited to: solution focused brief therapy, family therapy, cognitive-behavioural therapy; and education-based strategies. There is evidence in the literature to support the effectiveness of brief therapy; however, there is a lack of literature regarding its effectiveness in treating children s mental health issues in a community-based setting. This program evaluation served to promote evidence-based practice at CCTB. Professional development and training is a priority at CCTB as clinical staff regularly participate in training opportunities that complement interventions used in the Brief Services program, such as solution-focused brief therapy, family therapy, and cognitive behavioural therapy. The purpose of the evaluation was to formally evaluate, for the first time, the effectiveness of the Brief Services conducted at CCTB, to identify areas of strength and improvement of services, and to build CCTB s research-focused orientation and evaluation capacity. 4

5 Program Evaluation Questions Based on clinical judgment at Intake, were clients appropriately assigned to Brief as opposed to Long Term Services (i.e. clients with more severe presenting issues are assigned to Long Term Service)? What were the treatment goals and therapeutic strategies used in a typical Brief treatment session? Were clients satisfied with their Brief Service involvement? Was client satisfaction associated with greater improvement in treatment outcomes? Would the Brief Service program be effective in improving youth and adult mental health and overall functioning, as well as decreasing caregiver strain? Would therapeutic alliance be positively associated with treatment outcomes and client satisfaction? Would parent and youth depression, anxiety, and stress be associated with poorer treatment outcomes in youth mental health functioning and parent caregiver strain? Method Participants included clients who were assigned to brief outpatient treatment at CCTB. Clients included youth under the age of 18 years and their parent(s). A total of 82 clients were invited to participate. Fifty-six participants (68%) agreed to participate in the evaluation. At the end of the evaluation, 37 participants had all pre and post measures completed, which is a 66% completion rate. Participants completed measures at Intake, after each session, and upon completion of treatment and included measures of child functioning (The Brief Child and Family Phone Interview-3; BCFPI-3; Cunningham et al., 2006), adult mental health functioning (Depression, 5

6 Anxiety, and Stress Scale-21; DASS-21; Lovibond & Lovibond, 1995), Caregiver Strain (CGSQ; Brannan, Heflinger, & Bickman, 1997), therapeutic relationship (Working Alliance Inventory Short Form; WAI-S; Tracey & Kokotovic, 1989), and treatment practice elements. In order to determine if successful treatment outcomes could be attributed to Brief Service, the evaluation was conducted utilizing a pre/post research design. Specifically, families with children under the age of 18 years who would normally be assigned to Brief Service at CCTB were invited to participate. Normal exclusion criteria for the Brief Service program involved suicide ideation or attempts, sexual abuse or other significant trauma, grief issues, and having a significant co-morbidity (defined as having T-scores greater than 65 on multiple BCFPI-3 subscales). This service accepted children and families with a broad range of presenting issues, with treatment ranging in length from one to four sessions. Families were referred into this service by the Intake services program if they were determined to have mild to moderate level difficulties or could complete treatment within a maximum of four sessions. Brief Service clients participating in the evaluation consisted of 21 (57%) parents and 16 (43%) adolescents (i.e., age 12 or older). Eighteen youth (49%) were male and the age of youth ranged from one year seven months to 17 years and 9 months (with the average age being 11 years). New referrals made up 49% of the sample. The majority of Brief clients had a Moderate level of urgency with respect to their presenting issues (68%) and participated in four treatment sessions (38%). Data collection for this evaluation began May 9, 2011 and ended August 1, Specifically, a youth over 12 years of age receiving treatment without their parents completed the outcome measures themselves, including the DASS-21. However, youth did not complete the CGSQ as this questionnaire was a measure of caregiver strain. If a youth was under the 6

7 age of 12 years, or if they were 12 years or over and were joined by a parent in treatment, the parent completed all of the outcome measures for the child. Results, Discussion, and Interpretation Question 1: Based on clinical judgment at Intake, were clients appropriately assigned to Brief as opposed to Long Term Services (i.e. are clients with more severe presenting issues assigned to Long Term Service)? To address whether clients were appropriately assigned to Brief Services, scores on the pretreatment BCFPI-3 from both Brief and Long Term Service clients were compared, and no significant differences were found. A total of 80 clients participating in Long Term Service (defined as therapy longer than 4 sessions) at CCTB were included. Clients consisted of 37 adolescents (46%) and 43 parents (54%). The average age of children in Long Term Service was also 11 years. A total of 47% of the children in Long Term Service were male. Slightly more Long Term cases were new referrals (58%) in comparison to Brief clients. The majority of Long Term clients (60%) were rated as having an Urgent level of problem severity. When investigating possible differences between Brief and Long Term treatment cases on several key BCFPI-3 scales and demographic variables (i.e., age, gender, urgency level, and Internalizing, Externalizing, Managing Mood, Conduct, and Global Family Functioning scales), only urgency level significantly predicted entry into either Brief or Long Term Services. Specifically, Long Term clients (M = 2.84, SD =.68) had more urgent presenting issues than Brief clients (M = 3.58, SD =.50), as determined by Intake workers at the time of referral. Based on CCTB s service delivery model and current literature, it was expected that clients with less severe problems (indicated by lower BCFPI-3 scores) would be referred into the Brief Service program in comparison to clients triaged into regular Long Term outpatient treatment. 7

8 However, no significant differences were found between Brief and Long Term clients in terms of the severity of child mental health issues. Despite the relative similarity in child mental health issues between service programs, the majority of Long Term cases were rated as Urgent in terms of severity of presenting issue while the majority of Brief cases were rated as Moderate. This particular assessment of case severity was rated by the Intake worker using clinical judgment, while the BCFPI-3 described level of child behavioural and emotional problems as rated by the parent or youth. Thus, it appears that the classification of urgency by intake may be based on additional or different factors than that measured by the BCFPI-3. For example, the BCFPI-3 is child-focused and does not capture other important areas of functioning such as family relationships, life stress, poverty, and parent functioning. Alternatively, Intake workers may have different opinions of severity of presenting issues than that described by youth and their parents on self-report paper-and-pencil measures. Thus, if the presenting issues are relatively less severe, as provided by self-reports, it does not guarantee that they will be directed to Brief Service. It may be that other unknown factors, based on clinical judgment, are used to decide whether Brief or Long Term services are received. It is not clear if this is standardized or applied consistently across the therapists who make up the Intake team. If it is based solely on clinical judgment, there may be a risk of subjectivity and inconsistency in how severity is determined and which treatment program clients will then receive. Ultimately, further clarification of the criteria may be needed to ensure that clients are appropriately assigned to Brief and Long Term Services. 8

9 Question 2: What were the treatment goals and therapeutic strategies used in a typical Brief treatment session? To address this question, a profile of treatment goals and therapeutic strategies used in Brief Service was developed. The average session length was found to be minutes (SD = 15.48) and the overall session format or modality included: 43% family, 24% individual youth, and 33% parent formats. Parent-child conflict, dealing with high conflict separation/divorce, family relationship issues, and anxiety symptoms were the most common presenting issues for session one. Thus, the majority of Brief Service cases involved child internalizing issues and parenting concerns. Relationship/rapport building, family engagement, and supportive listening were the most commonly used therapeutic strategies used for session one. Lastly, emotional coping skills and reframing were consistently used across sessions two, three, and four. Question 3: Were clients satisfied with their Brief Service involvement? Was client satisfaction associated with greater improvement in treatment outcomes? Clients were very satisfied with their experiences at CCTB, with the majority of clients (58%) rating the overall quality of Brief Service as Excellent. In addition, 26% rated the overall quality as Very Good, 8% as Good, 5% as Average, and 3% as Fair. Clients indicated that Most Needs Were Met following their involvement with the Brief Services Program. Moreover, it was found that increased satisfaction scores were related to decreased reports of parent and youth depression and stronger client-rated therapeutic alliance. 9

10 Question 4: Would the Brief Service program be effective in improving youth and adult mental health and overall functioning, as well as decreasing caregiver strain? Pre/post data from relevant outcome measures were analyzed and it was found that significant changes were seen upon completion of Brief Service with respect to the following variables: BCFPI-3 Internalizing, Externalizing, and Total scales, DASS-21 Depression, Anxiety, Stress, and Total scales, and caregiver strain (Table 1 and Figure 1). The greatest improvements were seen on the Anxiety scale. The overall results are consistent with the existing literature on brief service, which suggests that it is effective, especially when targeting internalizing issues (e.g., Lee, 1997). Thus, there are numerous benefits for both children and their parents when participating in Brief Services. Question 5: Would therapeutic alliance be positively associated with treatment outcomes and client satisfaction? Therapeutic alliance was associated with better client satisfaction for sessions one and two as well as change in BCFPI-3 Total Mental Health score for session two. However, ratings of depression, anxiety, stress, and caregiver strain were not related to client-rated alliance for sessions one and two. Therapist ratings of alliance were not associated with any outcome measures. This highlights the importance of the client s perspective of the therapeutic alliance with respect to treatment outcomes. This is also consistent with the literature. For example, research evaluating the effectiveness of a manualized group-based parenting program at CCTB found that parent, but not therapist, ratings of alliance were associated with treatment outcomes (Schmidt, Chomycz, Houlding, Kruse, & Franks, under review). 10

11 More factors were expected to be associated with therapeutic alliance and positive treatment outcomes due to the fact that decreased mental health issues and caregiver strain, as well as increased therapeutic alliance, are all associated with positive treatment outcomes in the literature (e.g., Gunlicks & Weissman, 2008; Hawley & Weisz, 2005). However, it is important to note that the current literature on the connection between alliance and treatment outcomes pertains to Long Term Service. Furthermore, level of therapist and client agreement with respect to therapeutic alliance over time was investigated. Client and therapist ratings of alliance were related to each other for session two, but not session one. This suggests that therapeutic alliance builds over time and that only one Brief session may not be enough time for a strong or concordant therapeutic relationship to form. Question 6: Would parent and youth depression, anxiety, and stress be associated with poorer treatment outcomes in youth mental health functioning and parent caregiver strain? Ratings of parent and youth levels of depression, stress, and anxiety prior to Brief Service involvement were not related to changes in child treatment outcomes or changes in caregiver strain. This did not align with the existing literature which suggests that such factors as depression and stress play a role in treatment outcomes (e.g., Beauchaine et al., 2005; Brent et al., 1998). There is a possibility that the lack of significant results may be explained by the low sample size or lower levels of pre-treatment mental health issues. For example, only 13.1% of Brief Service cases reported severe or extremely severe levels of depressive symptoms prior to receiving service. Likewise, 28.3% reported similar levels of severe anxiety symptoms and 11

12 17.4% reported severe stress. This may also suggest that other factors play more of a role in predicting treatment change, both prior to and during treatment, such as type of presenting issue. However, follow-up studies are required to further investigate the strength of the relationship between parent psychopathology and treatment outcomes. Conclusions In summary, the CCTB Brief Service program was found to be effective in improving child and parent mental health functioning and caregiver strain. Clients were also satisfied with their overall experiences at CCTB. Thus, Brief Service can be beneficial in community-based clinics for treating a variety of presenting concerns. Factors that contribute to successful outcomes in therapy were also identified, which can benefit mental health agencies providing Brief Services. Several limitations of this evaluation must be noted. Primarily, clients varied with respect to the number of sessions they engaged in and the type of treatment modalities that were used during the Brief Service, thereby providing a treatment confound. Moreover, there was no experimental control of service delivery in order to control for the passage of time as a factor in symptom improvements. Additionally, the evaluation results were limited by the small sample size. Dissemination of results to staff, stakeholders, and other community-based clinics with similar service initiatives is currently planned. Likewise, CCTB plans to continue to promote evidence-based services by evaluating additional programs and further validating their effectiveness. 12

13 References Beauchaine, T. P., Webster-Stratton, C., & Reid, M. J. (2005). Mediators, moderators, and predictors of 1-year outcomes among children treated for early-onset conduct problems: A latent growth curve analysis. Journal of Consulting and Clinical Psychology, 73, doi: / X Brannan, A. M., Heflinger, C. A., & Bickman, L. (1997). The Caregiver Strain Questionnaire: Measuring the impact on the family of living with a child with serious emotional disturbance. Journal of Emotional and Behavioural Disorders, 5, doi: / Brent, D. A., Kolko, D. J., Birmaher, B., Baugher, M., Bridge, J., Roth, C., & Holder, D. (1998). Predictors of treatment efficacy in a clinical trial of three psychosocial treatments for adolescent depression. Journal of the American Academy of Child and Adolescent Psychiatry, 37, doi: / Cunningham, C. E., Pettingill, P., & Boyle, M. (2006). The Brief Child and Family Phone Interview (BCFPI-3): A computerized intake and outcome assessment tool. Interviewers manual. Hamilton, ON: Offord Centre for Child Studies, McMaster University. Gunlicks, M. L., & Weissman, M. M. (2008). Change in child psychopathology with improvement in parental depression: A systematic review. Journal of the American Academy of Child and Adolescent Psychiatry, 47, doi: / CHI.0b013e

14 Hawley, K. M., & Weisz, J. R. (2005). Youth versus parent working alliance in usual clinic care: Distinctive associations with retention, satisfaction, and treatment outcome. Journal of Clinical Child and Adolescent Psychology, 34, doi: / s jccp3401_11 Lee, M. (1997). A study of solution-focused brief family therapy: Outcomes and issues. The American Journal of Family Therapy, 25, doi: / Lovibond, P. F., & Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scale (DASS) with the Beck Depression and Anxiety Inventories. Behaviour Research and Therapy, 33, doi: / (94)00075-U Schmidt, F., Chomycz, S., Houlding, C., Kruse, A., & Franks, J. (under review). The association of therapeutic alliance with treatment outcomes for a group Triple P intervention. Manuscript submitted for publication. Tracey, T. J., & Kokotovic, A. M. (1989). Factor structure of the Working Alliance Inventory. A Journal of Consulting and Clinical Psychology, 1, doi: /

15 Figure 1 Comparison of Pre/Post Treatment Results (n = 37) Pre Post Measure 15

16 Table 1 Summary of Effectiveness on Outcome Measures (n = 37) Pre-Tx Post-Tx Measure M SD M SD df t ES (r) BCFPI-3 Total *.42 Externalizing *.44 Internalizing *.52 DASS-21 Depression **.59 Stress **.69 Anxiety ***.84 Total Score ***.83 Caregiver Strain *.33 Note. An effect size of.2 is considered small,.5 is moderate, and.8 is large. *p <.05. **p <.01. ***p <