A Comparison of Children In-Care. And. Children Not In-Care. As Measured by Standardized Assessment Tools

Transcription

1 A Comparison of Children In-Care And Children Not In-Care As Measured by Standardized Assessment Tools March 30, 2007 Andre Breton Patricia Centre for Children a Youth

2 2 Funding for this project was provided by the Provincial Centre of Excellence for Child and Youth Mental Health at the Children's Hospital of Eastern Ontario. The following individuals from the Patricia Centre for Children and Youth are acknowledged for their contributions to the completion of this project: Tricia England Lori Kutney Debbie Dokuchie Janet Paterson Maureen Sullivan Pam De Ridder Sheila Fenelon Linda Leochko Judy Wallace The thoughts contained in this report are those of the author and may not be similarly shared by The Provincial Centre of Excellence for Child and Youth Mental Health at the Children's Hospital of Eastern Ontario.

3 3 EXECUTIVE SUMMARY...5 PURPOSE OF EVALUATION...7 BACKGROUND...8 DESCRIPTION OF THE PROGRAM/PROJECT TO BE EVALUATED AND TARGET POPULATION...8 RELEVANT STAKEHOLDERS...8 REVIEW OF RELATED RESEARCH...9 METHODOLOGY...10 DESIGN OF EVALUATION...10 Clients at Intake/Assessment...10 Clients at Closure...12 SOURCES OF INFORMATION...14 Brief Child and Family Phone Interview (BCFPI)...14 Child and Adolescent Functional Assessment Scale (CAFAS)...18 DATA ANALYSIS AND STATISTICAL PROCEDURES...20 BCFPI Scores (Intake)...20 CAFAS Scores (Assessment)...20 CAFAS Scores (Closure)...20 EVALUATION LIMITATIONS...20 RESULTS...23 CLIENTS AT INTAKE...23 Comparison of Means of BCFPI Scores on Each Subscale...23 Means of Externalizing Mental Health Subscales Means of Internalizing Mental Health Subscales Means of Composite Scales Means of Child Functional Impact Scales Means of Family Functional Impact Scales Summary of Results of Comparison of BCFPI Means Comparison of the Prevalence of Clinically BCFPI Scores on Each Subscale...32 Prevalence of Externalizing Mental Health Subscales Prevalence of Internalizing Mental Health Subscales Prevalence on Composite Scales Prevalence on Child Functional Impact Scales Prevalence on Family Functional Impact Scales Prevalence of At Least One Clinically Score Prevalence of Two or More Clinically Scores Summary of Results of Comparison of BCFPI Prevalence Scores CLIENTS AT ASSESSMENT...43 Comparison of Means on CAFAS Scales...44 Central Tendency Subscale Means at Entry Summary of Comparison of Mean CAFAS Scores COMPARISON OF PREVALENCE OF MODERATE AND SEVERE IMPAIRMENT ON CAFAS SUBSCALES...47 Comparison of Moderate and Severe Impairment on 0 to 8 Subscales...47 School Role Subscale...49 Home Role Subscale...50 Community Role Subscale*...51 Behaviour Towards Others Subscale...52 Moods/Emotions Subscale...53 Self Harm Subscale*...54 Substance Use Subscale*...55 Thinking Subscale...56

4 4 Summary of Comparison of Prevalence of Moderate and Severe Impairment on Subscales:...56 CLIENTS AT CLOSURE...57 COMPARISON OF CAFAS MEANS PRE AND POST SERVICE...57 Measures of Central Tendency at Entry and Exit to Service:...57 CAFAS Mean School Role Subscale Pre and Post Service...58 CAFAS Mean Home Role Subscale Pre and Post Service...59 CAFAS Mean Community Role Performance Pre and Post Service...60 CAFAS Mean Behaviour Towards Others Pre and Post Service...61 CAFAS Mean Moods/Emotions Subscale Pre and Post Service...62 CAFAS Mean Self Harm Subscale Pre and Post Service...63 CAFAS Mean Substance Use Subscale Pre and Post Service...64 CAFAS Mean Thinking Subscale Pre and Post Service...65 Summary of Comparison of Pre and Post Means...66 COMPARISON OF CAFAS PROPORTION OF CLIENTS IMPROVED...67 CAFAS Overall Impairment at Entry and Exit...67 CAFAS TIERS at Exit...68 CAFAS Clinically Meaningful Reduction in Overall Impairment...69 CAFAS Absolute Change in Level of Functioning...70 Serious Emotional Disturbance Based on Total Score...71 Serious Emotional Disturbance Based on Subscale Scores...72 Summary of Comparison of Proportion of Clients Improved...73 CONCLUSIONS AND RECOMMENDATIONS...74 DISCUSSION AND INTERPRETATION OF FINDINGS...74 Brief Child and Family Phone Interview (Intake)...74 CAFAS (Assessment)...76 CAFAS (Closure)...77 CONCLUSIONS AND RECOMMENDATIONS...78 LESSONS LEARNED FROM EVALUATION ACTIVITIES...80 IMPACT ON CLIENTS SERVED, STAFF AND THE ORGANIZATION AS A WHOLE...80 NEXT STEPS...81 KNOWLEDGE EXCHANGE PLAN...81 OVERVIEW OF KNOWLEDGE EXCHANGE ACTIVITIES...81 KNOWLEDGE EXCHANGE ACTIVITIES ACCOMPLISHED TO DATE...81 FURTHER PLANS REGARDING KNOWLEDGE EXCHANGE ACTIVITIES...81

5 5 EXECUTIVE SUMMARY This evaluation had two main objectives. The first is to determine whether the level of impairment and symptom severity at Intake and Assessment of children and youth in the care of children s aid societies was significantly different than that of children not in the care of children s aid societies. The second objective of the evaluation was to determine whether the outcomes at Closure of children and youth in the care of children s aid societies were significantly different than those of children and youth not in the care of children s aid societies. The specific questions addressed in this evaluation were the following: a) Are the average (mean) Brief Child and Family Phone Interview (BCFPI) subscale scores of children in care significantly different than those of children not in care? b) Is the prevalence of clinically significant scores on the BCFPI subscales of children in care significantly different than that of children not in care? c) Are the average (mean) Child and Adolescent Functional Assessment Scale (CAFAS) subscale and Total Scores at Assessment of children in care significantly different than those of children not in care? d) Is the prevalence of Moderate and Severe CAFAS subscale scores at Assessment of children in care significantly different than that of children not in care? e) Are the changes in average (mean) CAFAS scores from treatment Entry (T1) to treatment Exit (T14) significantly different for children in care of children s aid societies as compared to children not in care? f) Are the proportions of clients improved from treatment Entry (T1) to treatment Exit (T14) on various CAFAS outcome measures significantly different for children in care of children s aid societies as compared to children not in care? The evaluation compared BCFPI profiles and CAFAS profiles of clients who had received service in the Child and Family Intervention between October 1, 2003 and September 30, The clients were divided into two groups: a) those that were in the care of a children s aid society at the commencement of services, and b) those that were not in the care of a children s aid society at the commencement of services. The comparison of the two groups yielded the following findings: 1. Comparison of mean scores on the BCFPI Mental Health and Composite subscales (total 12 subscales) revealed only two statistically significant differences, those being on the Conduct and Managing Mood subscales. However, there were statistically significant differences on six of the seven Functional Impact Scales (Social Participation, Quality of Relationships, School Participation and Achievement, Global Child Functioning, Family Comfort, and Global Family Situation).

6 6 2. Comparison of the prevalence of clinically significant scores on the BCFPI Mental Health and Composite subscales (total 12 subscales) revealed only one statistically significant difference, that being on the Cooperativeness subscale. However, there were statistically significant differences on five of the seven Functional Impact Scales (Social Participation, School Participation and Achievement, Global Child Functioning, Family Comfort, and Global Family Situation). 3. Comparison of the means of CAFAS Total scores and CAFAS Subscale scores (total 9 comparisons) at Entry (T1) revealed only one statistically significant difference, that being on the School Role subscale. 4. Comparison of the prevalence of Moderate and Severe scores on the eight CAFAS subscales at Entry (T1) revealed three statistically significant differences, those being on the School Role, Self Harm, and Substance Use Subscales. 5. Comparison of the changes in mean CAFAS scores (Total score and 8 subscale scores) from Entry (T1) to Exit (T14) of both groups revealed only two differences between the groups. Only the Not In Care group evidenced statistically significant decreases on the Home Role and Thinking subscales. 6. Comparison of proportions of clients improved from Entry (T1) to Exit (T14) using six different CAFAS outcome measures revealed no statistically significant differences between the proportions of clients that improved in both groups. The results of the comparison of BCFPI scores suggest that, as recommended by Cunningham et al (2006), those using and interpreting the BCFPI should go beyond considering the numerical results only. They should be aware of and consider the limitations to the interpretation of the BCFPI (p.26-33) and interpret the instrument using the 22 steps outlined by the authors (p.33-39). The results do not support favouring one group over the other in triaging clients for priority service. The results of the comparison of CAFAS Entry scores revealed that while the Not In Care group had a higher mean School Role subscale score, and higher prevalence of Moderate and Severe subscale scores on the School Role, Self Harm and Substance Use subscales, there was no statistically significant difference in Mean Total score. The results suggest the Not In Care group had a higher proportion of clients in the Self Harm Potential and Maladaptive Substance Use CAFAS TIERS, the second and third most severe TIERS. The results of the comparison of CAFAS Pre/Post scores revealed that while only the Not In Care group evidenced statistically significant decreases on the Home Role and Thinking subscales, both groups evidenced a statistically significant decrease in CAFAS Total score. Further, the comparison of proportions of clients improved from Entry (T1) to Exit (T14) using six different CAFAS outcome measures revealed no statistically significant differences between the proportions of clients improved in both groups. The results indicate that both groups benefit from the program.

7 7 PURPOSE OF EVALUATION This evaluation had two main objectives. The first is to determine whether the level of impairment and symptom severity at Intake and Assessment of children and youth in the care of children s aid societies was significantly different than that of children not in the care of children s aid societies. Children in care refers to children who were crown wards, society wards, or children in temporary care or customary care of a children's aid society at the beginning of service with the Patricia Centre for Children and Youth. This is relevant to the agency operations, since as most children s mental health centres in the Province, the agency has a waiting list for service. The agency must therefore prioritize cases for service. If one group of clients was consistently found to have a significantly higher level of impairment or symptom severity at intake, client status (In Care, Not In Care) could be used as one factor in prioritizing cases for service. The following questions relate to the first objective: a) Are the average (mean) Brief Child and Family Phone Interview (BCFPI) subscale scores of children in care significantly different than those of children not in care? b) Is the prevalence of clinically significant scores on the BCFPI subscales of children in care significantly different than those of children not in care? c) Are the average (mean) Child and Adolescent Functional Assessment Scale (CAFAS) subscale and Total Scores at Assessment of children in care significantly different than those of children not in care? d) Is the prevalence of Moderate and Severe CAFAS subscale scores at Assessment of children in care significantly different than that of children not in care? The second objective of the evaluation was to determine whether the outcomes at Closure of children and youth in the care of children s aid societies are significantly different than those of children and youth not in the care of children s aid societies. This is relevant to agency operations since if outcomes for one group were significantly better, the reasons for better outcomes could be sought. These findings could then possibly be used to improve the outcomes for the second group. The following questions relate to the second objective: e) Are the changes in average (mean) CAFAS scores from treatment Entry (T1) to treatment Exit (T14) significantly different for children in care of children s aid societies as compared to children not in care?

8 8 f) Are the proportions of clients improved from treatment Entry (T1) to treatment Exit (T14) on various CAFAS outcome measures significantly different for children in the care of children s aid societies as compared to children not in care? BACKGROUND Description of the Program/Project to be Evaluated and Target Population The Patricia Centre for Children and Youth is an accredited children s mental health centre which assists families with children who are experiencing emotional, behavioural, developmental or interpersonal difficulties. The agency is a non-profit corporation governed by a volunteer Board of Directors, and funded primarily by the Province of Ontario. The agency s administrative office is located in Dryden, Ontario, and services are provided from offices located in Dryden (Team Central), Red Lake (Team West) and Sioux Lookout (Team North), Ontario. The program that was subject to this evaluation is the Child and Family Intervention Program. The program addresses a wide array of emotional, behavioural and psychosocial problems experienced by children, youth and their families. The intent of the program is to help children, youth (age 0-18) and their families develop skills and build competencies that will lead to selfsufficiency and independence, rather than long-term reliance on the system. Services are delivered through geographically based Teams (as noted above). Each Team is staffed by Counsellors with a range of educational backgrounds (ex: social work, nursing, child and youth work, early childhood education, psychology, social service worker). Each Team of Counsellors is lead by a Team Leader who is responsible for supervision of program delivery. The services provided include individual, group and family counselling, consultation, parent education and case management. Counselling is based upon an individualized service plan which considers the holistic needs of the client and is mutually developed with the client/family. The program is funded by the Ontario Ministry of Children and Youth Services/Ministry of Community and Social Services, and has an annual budget of approximately $1,000,000. Relevant Stakeholders The relevant stakeholders for this evaluation include the following: Ministry of Children and Youth Services/Ministry of Community and Social Services: As noted above, the Ministry provides annual funding for the program through transfer payments. Board of Directors, Patricia Centre for Children and Youth: The volunteer Board of Directors contracts with the Ministry to provide a variety of programs, including the Child and Family Intervention program. In addition, to its contractual and financial responsibility for the program, the Board acts as stewards for the agency and represents the interests of the community.

9 9 Management and Staff, Patricia Centre for Children and Youth: The management and staff of the agency are responsible for program delivery within the terms of the service contract with the Ministry, related legislation and standards of quality as established by Children s Mental Health Ontario. Clients: The clients of the program, those in care and not in care, are also stakeholders in this evaluation. Insufficient funding results in waiting lists for services. Clients have a vested interest in knowing that triaging decisions made by the agency are well informed. Clients also have a vested interest in knowing the outcomes of the services provided. Affiliated Agencies: The children s aid societies who refer children in care to the program are also stakeholders of this evaluation. As noted above, insufficient funding results in waiting lists, and children in care may also therefore have to wait for services. These children and youth, and their guardians have a vested interest in knowing that triaging decisions made by the agency are well informed, and in knowing the outcome of services provided. Review of Related Research A search of available research did not reveal studies which compared these groups of clients using the BCFPI and the CAFAS. A related study by Michael O Brien (2006) described the use and results of the CAFAS for rating children in care at Family and Children s Services of Renfrew County. At that agency, the CAFAS is completed at the time of admission, and then every six months as long as the child remains in care. Among the findings reported by O Brien, are: a) that after an improvement seen at the time the CAFAS is administered for the second time, the pattern is to then regress towards the initial score over time; b) the most serious scores are found in the school, home, behaviour towards others, and moods/emotions sub-scales; c) a small amount of improvement occurred in the school, moods/emotions and thinking sub-scales; d) the entry scores of the children in care were similar to those of outpatient and inpatient cases in the children s mental health system as reported by the Hospital for Sick Children in 2005; e) while considerable improvement occurred in the children s mental health population, this was not the case for the children in care f) children over the age of 12 were found to have improved over time, whereas children under 12 deteriorated. A related study reported by Flynn (2003) compared level of developmental outcomes and resilience among young people in out-of-home care in CAS s in Ontario. The study compared a sample of 503 children in care in Ontario to a national sample of youths from the general Canadian population on 7 dimensions (health, education, identity, social presentation, family and social relationships, emotional and behavioural development, and self-care skills). Among the findings was that there were many positive outcomes and resilience among children in care (e.g. pro-social behaviours, self-esteem, optimism about the future), but priority areas for improvement were education and behavioural and emotional development. More than half of the

10 10 children in care were rated as not resilient in the area of level of anxiety/emotional distress (compared to 27% for the national sample), and more than 60% were rated as not resilient in the area of level of hyperactivity/inattention (33% for the national sample), more than 70 percent were rated as not resilient in the area of how the child was doing in school (34% for national sample). METHODOLOGY Design of Evaluation The evaluation sought to compare two groups of clients who received service in the Child and Family Intervention program: clients who were in the care of a children's aid society at the start of service, and clients who were not in the care of a children's aid society at the start of service. The groups would be compared using two instruments: the Brief Child and Family Phone Interview Parent Version (BCFPI) and the Child and Adolescent Functional Assessment Scale (CAFAS). The comparison of scores at Intake/Assessment would be for clients who began service between October 1, 2003 and September 30, The comparison of Scores at Closure would be for clients who ended service between October 1, 2003 and September 30, Given that the evaluation began in December, 2006, the evaluation was based on data which already existed within the agency, that being BCFPI profiles (Parent Version) and CAFAS profiles. The BCFPI profiles (hard copy only) had been received by the agency from Integrated Services Northwest (ISN), the agency which acts as the single point of access for the Patricia Centre for Children and Youth. The CAFAS profiles had been completed by the staff of the Patricia Centre for Children and Youth. Clients at Intake/Assessment The profiles selected for the clients at intake portion of this evaluation were the following: admission date into the agency between October 1, 2003 and September 30, 2006, and, CAFAS T1 (1 st Evaluation) completed between October 1, 2003 and September 30, 2006 A search of the agency CAFAS data base yielded a total sample size of 346 clients who met the above-noted criteria. The CAFAS profiles were downloaded into an Excel file for analysis. Once these clients had been identified, the corresponding BCFPI (Parent Version) profiles were retrieved from the client files (as noted earlier, ISN only provides hard copies of the BCFPI profiles). Only the Parent Version of the BCFPI was used. This decision was in part based on the limited resources (time and funding) available to complete the project. However, as noted by CMHO (2007),... adolescents consistently rate themselves as being less troubled than adults in their lives rate them (slide 13). The clients were then divided into two distinct groups: a) those that were in the care of a CAS at the time of admission, and b) those that were not in care at the time of admission. This was accomplished using the following steps:

11 11 a search of the agency s MIS system was conducted on each client to determine client status at intake (i.e. In Care or Not In Care). in order to ensure that the information in the MIS system was correct, all 346 client files (hard copy, or electronic copy if the hard copy had been archived) were reviewed to confirm client status (i.e. In Care or Not In Care). Table 1 illustrates the sample of Clients at Intake/Assessment, the number of CAFAS (T1) profiles, and the number of BCFPI profiles. Table 1: Evaluation Sample Clients at Intake/Assessment: Clients In Care (N= 64) Clients Not In Care (N = 282) TOTAL (N = 346) N % N % N % CAFAS T1 (Entry) % % % BCFPI % % % Client status at Intake by geographic Team is illustrated in Figure 1 below. Figure 1 Client Status at Intake % 90.00% 80.00% 73.45% 89.53% 90.36% 81.50% 70.00% % of Clients 60.00% 50.00% 40.00% 30.00% 26.55% 20.00% 18.50% 10.00% 0.00% 10.47% 9.64% Central North West Total In Care Not In Care The clients who were In Care at the time of Intake came from three different children s aid societies. This is illustrated by geographic Team in the Figure 2 below:

12 12 Figure 2 Clients in Care by Children's Aid Society and Location # of Clients Central North West Total CAS 1 CAS 2 CAS 3 Total Clients at Closure The profiles selected for the clients at closure portion of this evaluation were the following: closure date from the agency between October 1, 2003 and September 30, 2006 (regardless of the date of admission); CAFAS T1 (First Evaluation) on file, regardless of date of Entry into the program, and; CAFAS T14 (Exit Evaluation) completed between October 1, 2003 and September 30, 2006 Note that this is NOT the same group of clients as the intake group, but there is some overlap between the groups. A search of the agency CAFAS data base yielded a total sample size of 324 clients who met the above-noted criteria. The CAFAS profiles were downloaded into an Excel file for analysis. The clients were then divided into two distinct groups: a) those that were in the care of a CAS at the time of admission, and b) those that were not in care at the time of admission. This was accomplished using the following steps:

13 13 a search of the agency s MIS system was conducted on each client to determine client status at intake (i.e. In Care or Not In Care). in order to ensure that the information in the MIS system was correct, all 324 client files (hard copy, or electronic copy if the hard copy had been archived) were reviewed to confirm client status (i.e. In Care or Not In Care). Table 2 illustrates the sample of Clients at Closure and the number of CAFAS Entry (T1) profiles, and the number of CAFAS Exit (T14) profiles. Table 2: Evaluation Sample Clients at Closure: Clients In Care (N= 64) Clients Not In Care (N = 260) TOTAL (N = 324) N % N % N % CAFAS T1 (Entry) % % % CAFAS T14 (Exit) % % % Client status by geographic Team is illustrated in Figure 3 below. Figure 3 Client Status by Team % 90.00% 80.00% 74.53% 86.96% 85.11% 80.25% 70.00% % of Clients 60.00% 50.00% 40.00% 30.00% 20.00% 25.47% 13.04% 14.89% 19.75% 10.00% 0.00% Central North West Total Team In Care Not In Care

14 14 The clients who were In Care in this sample came from three different children s aid societies. This is illustrated by geographic Team in Figure 4 below. Figure 4 Clients in Care by Children's Aid Society and Location # of Clients Central North West Total CAS 1 CAS 2 CAS 3 Total Sources of Information As noted previously, the sources of data for the evaluation were the BCFPI and the CAFAS. A brief description of each instrument and how it was implemented is presented below. Brief Child and Family Phone Interview (BCFPI) The following description of the BCFPI is taken from ''The Brief Child and Family Phone Interview (BCFPI-3), A Computerized Intake and Outcome Assessment Tool, Interviewers Manual, October, 2006 Edition''; Charles E. Cunningham, Peter Pettingill, Michael Boyle. The BCFPI is a required core intake tool of children's mental health services in Ontario, Canada (p.63). The Brief Child and Family Phone Interview (BCFPI) is an abbreviated (e.g. 30 minute) standardized or structured interview administered by phone to parents, teachers, or adolescents (p. 5). The BCFPI is completed at the point of intake prior to clinical assessment and treatment. The BCFPI may be administered during the first phone contact with a client or scheduled for a time that is convenient for the family (p. 5). The BCFPI is typically administered to the parents and teachers of 3 to 18 year olds. An adolescent self-report interview is available for youth aged

15 15 12 to 18 (p. 6). The BCFPI's parent, adolescent and teacher interviews may be administered in person with the interviewer recording responses on a paper version of the BCFPI. Alternatively, individual clients or groups of clients can complete a paper and pencil version of the BCFPI. Responses can be entered and scored by the BCFPI software at a later point in time (p. 7). The BCFPI is not a diagnostic tool. The BCFPI provides descriptive information that does not reflect assumptions regarding the etiology or causes of a child's problems (p.26). The BCFPI's questions are designed to screen for common referral concerns, estimate their impact on child and family functioning, consider interim service options, and anticipate barriers that might prevent a family from using a potentially helpful services. Like all standardized measurement tools (and clinical interviews), the BCFPI will yield both false positive and false negative results. The BCFPI should not, therefore, be used as a stand-alone screening, triaging, or treatment decision-making tool (p. 27). The BCFPI's Standard Parent Report compares an individual child's score to a random population sample of children from the Ontario Child Health Study's revised measurement project (Boyle et al, 1993a). Population norms should be used when interpreting the BCFPI (p. 28). The results of the BCFPI are summarized as t-scores. T scores are standardized measures based on a distribution with a mean of 50 and a standard deviation of 10. The average score for the population on which the score is based is 50. A t-score of 50 corresponds to a percentile score of 50. The scores of 50% of the population are lower than a t-score of 50. The scores of approximately 84% of the population are lower than a t-score of 60. The scores of approximately 93% of the population are below at t score of 65. The scores of approximately 98% of the population are lower than a t score of 70 (p.29). The reliability and validity of the BCFPI are well established (Cunningham et al, 2006, p ). In the catchment area served by the Patricia Centre for Children and Youth, the BCFPI is administered by an agency called Integrated Services Northwest (ISN). ISN is an agency created by the Ministry of Community and Social Services/Ministry of Children and Youth Services (Ministry) in One of the functions given to ISN by the Ministry was to be the single point of access to the Patricia Centre for Children and Youth. ISN therefore administers the BCFPI, and provides hard copies of the reports to the Patricia Centre. In this evaluation, only the Parent Version of the BCFPI was used. The two groups were compared on the following 19 subscales. Regulating Attention This describes the child's ability to sustain attention, complete tasks, and avoid distractions. High scores on this subscale correspond to the types of problems experienced by children with the predominantly inattentive type of ADHD as described in the DSM-IV (Cunningham et al, October, 2006, p. 29).

16 16 Regulating Impulsivity and Activity Level* This describes the child's ability to regulate activity level and impulsive responding. High scores on this subscale correspond to the types of problems experienced by children with the predominantly impulsive-hyperactive type of ADHD as described in DSM-IV (Cunningham et al, October, 2006 p. 29). Regulating Attention, Impulsivity and Activity Level This subscale is composed of the three Regulating Attention questions and the three Regulating Impulsivity and Activity questions. High t-scores on this scale reflect over active and impulsive behaviour. The items on this subscale correspond to the types of problems evidenced by children with the combined type of ADHD described in DSM-IV (Cunningham et al, October, 2006, p ). Cooperativeness with Others This subscale is composed of items reflecting the extent to which the child is engaged in cooperative relationships with others. High t-scores reflect non-compliant, defiant, resentful relationships with adults and peers. These behaviours correspond to Oppositional Defiant Disorder in the DSM-IV (Cunningham et al, October, 2006, p.30). In Care: N = 56, Mean: 60.29, Percentile Rank: 84.1 Conduct This subscale reflects serious rule violations and antisocial behaviour. Because the items on this scale occur infrequently in nonclinical normative populations, high t scores will result when a small number of items are endorsed or several items are endorsed at a low level. These questions correspond to the Conduct Disorder Scale in DSM-IV (Cunningham et al, October, 2006, p.30). Separating from Parents This subscale reflects the extent to which the child is able to separate comfortably from parents. High t-scores reflect difficulties separating from parents and correspond to Separation Anxiety Disorder in the DSM-IV (Cunningham et al, October, 2006, p. 30). Managing Anxiety This scale is composed of items reflecting the extent to which the child worries about past, present and future events. High t-scores on this subscale reflect difficulties with anxiety and correspond to Anxiety Disorder in DSM-IV (Cunningham et al, October, 2006, p.30). Managing Mood This subscale is composed of questions reflecting interest or enjoyment of life and general mood. High t-scores on this scale suggest that the child may be losing interest in activities and

17 17 relationships which have previously been a source of pleasure (Cunningham et al, October, 2006, p. 30). Self Harm The subscale reflects concerns regarding weight loss, suicidal talk or suicidal attempts. This subscale may be administered when elevated scores on the Managing Mood subscale are reported or routinely to all children. The self harm score is based on these 3 self harm questions plus the previous 6 Managing Mood questions. This 9 item composite scale corresponds to the DSM-IV's Major Depressive Episode (Cunningham et al, October, 2006, p. 30). Externalizing Behaviour This 18 item scale is composed of the 6 item Regulating Attention, Impulsivity and Activity Level subscale, the 6 item Cooperativeness subscale, and the 6 item Conduct subscale. An Externalizing score is computed only if all 3 BCFPI externalizing subscales have been computed (Cunningham et al, October, 2006, p. 30). Internalizing Behaviour This 18 item scale is composed of the 6 item Separation from Adults subscale, the 6 item Managing Anxiety subscale, and the 6 item Managing Mood subscales. An Internalizing score is computed only if all 3 BCFPI internalizing subscales have been computed (Cunningham et al, October, 2006, p. 31). Total Problems This 36 item scale is composed of the 18 item Externalizing Problems Scale and the 18 item Internalizing Problems Scale. A total problem score is computed only if all 6 BCFPI Mental Health Scales have been computed (Cunningham et al, October, 2006, p.31). Child' Social Participation High t-scores on this subscale suggest that the child may be withdrawing or spending less time with other children (Cunningham et al, October, 2006, p. 31). Quality of the Child's Social Relationships High t-scores reflect poor relationships with parents, teachers or peers (Cunningham et al, October, 2006, p. 31). School Participation and Achievement This subscale reflects school attendance and grades. The score for this scale is based on a question regarding attendance, a question regarding grades, and a 3 rd question regarding

18 18 relationships with teachers. High t-scores suggest attendance problems, academic difficulties and or poor relationships with teachers (Cunningham et al, October, 2006, p. 31). Global Child/Youth Functioning This composite scale combines questions from the Social Participation, Quality of Relationships, and School Performance/Achievement Subscales described above. It provides a global estimate of Child Functioning with higher scores reflecting more impairment. (Cunningham et al, October, 2006, p. 31). Family Activities This subscale reflects the extent to which the child's problems are perceived to have influenced the family's external networks. These include the extent to which the child's behaviour influences visits from friends and relatives, the family's ability to use child care, and the family's ability to take the child on shopping trips or visits. High t-scores on this subscale suggest that the child is perceived to limit the family's relationships with friends and family and/or the family's mobility in the community (Cunningham et al, March, 2002, p. 28). Family Comfort This subscale reflects the perceived impact of the child's problems on more internal family functioning. High t-scores on this subscale suggest that the child is perceived to be a source of conflict and anxiety in the family (Cunningham et al, March, 2002, p. 28). Global Family Situation This composite scale combines items from the Family Activities and Family Comfort scales (Cunningham et al, March, 2002, p. 28). This score provides an overall estimate of the impact of the problems discussed in the BCFPI interview on family functioning. Higher t-scores reflect higher overall levels of functional impairment and risk. This score is, again, important in understanding the contextual factors which may influence service planning and outcome (Cunningham et al, March, 2002, p ). Child and Adolescent Functional Assessment Scale (CAFAS) The following description of the CAFAS is taken from ''CAFAS, Manual for Training Coordinators, Clinical Administrators, and Data Managers'', Second Edition, 2003, Kay Hodges, Ph.D., p. 1-8). The purpose of the Child and Adolescent Functional Assessment Scale (CAFAS) is to measure impairment in day-to-day functioning in children and adolescents. The CAFAS was designed to assess impairment in children and adolescents who have or are at risk for emotional, behavioural, substance abuse, psychiatric, or psychological problems. The CAFAS provides for rating the youth across different domains of functioning, and, for each domain, along a continuum of impairment.

19 19 The CAFAS is not administered. The CAFAS can be rated by a professional/staff member who is well informed about the child and is a reliable rater on the CAFAS. The rater does not assign a score; rather, the items endorsed by the rater determine the score. Subscales Assessing Youth School/Work Role Performance: Ability to function satisfactorily in a group educational environment Home Role Performance: Extent to which youth observes reasonable rules and performs age appropriate tasks Community Role Performance: Respect for the rights of others and their property and conformity to laws Behaviour Toward Others: Appropriateness of the youth s daily behaviour Moods/Emotions: Modulation of the youth s emotional life Self-Harmful Behaviour: Extent to which the youth can cope without resorting to self-harmful behaviour or verbalizations Substance Use: Youth s substance use and the extent to which it is not appropriate or disruptive Thinking: Ability of youth to use rational thought processes Levels of Impairment: Within each subscale, the behavioural items are grouped by severity into four columns, using a continuum of severity of impairment. The four levels of severity and the associated assigned scores are as follows: Severe Impairment: Severe disruption or incapacitation (30) Moderate Impairment: Major or persistent disruption (20)

20 20 Mild Impairment: problems or distress (10) Minimal or No Impairment: No disruption of functioning (0) After marking items that apply to the youth, the subscale scores are apparent. The total is the sum of the 8 subscale scores for the youth. The higher the score, the greater the impairment. The reliability and validity of the CAFAS are well established (Hodges, 2003, p ). All agency staff who rated the CAFAS were certified in Inter-Rater Reliability by CAFAS in Ontario/The Hospital for Sick Children. Data Analysis and Statistical Procedures BCFPI Scores (Intake) Prior to using the t-test (two tail) to compare the two groups, an F-test for two sample variances was performed on each of the scales/subscales to be compared. This was necessary in order to determine whether the variances were equal or unequal. The results of the F-tests indicated that for the majority of comparisons (15 of 19 subscales), a t-test for two samples assuming equal variances was appropriate. In a minority of cases (Regulating Impulsivity and Activity, Conduct, Managing Mood, Total Problems), a t-test for two samples assuming unequal variances was appropriate. The latter instances are indicated by an asterisk (*) next to the subscale titles. CAFAS Scores (Assessment) Prior to using the t-test (two tail) to compare the two groups, an F-test for two sample variances was performed on each of the scales/subscales to be compared. This was necessary in order to determine whether the variances were equal or unequal. The results of the F-tests indicated that for the majority of comparisons, a t-test for two samples assuming equal variances was appropriate. In a minority of cases (Community Role, Self Harm, and Substance Use), a t-test for two samples assuming unequal variances was appropriate. The latter instances are indicated by an asterisk (*) next to the subscale titles. CAFAS Scores (Closure) To compare the changes in mean scores from Entry to Exit, each group was compared using the t-test, Paired Two Sample for Means. To compare the proportion of clients improved, a t-test (two tail) for two samples assuming equal variance was used. Evaluation Limitations The following limitations of the evaluation were identified:

21 21 a) Measurement Instruments: As noted above, the measurement instruments used in this evaluation were the BCFPI and CAFAS. These instruments have their own inherent limitations. Readers are referred to Cunningham et al, 2006, and Hodges, 2003 Second Edition for a complete description of those limitations. b) Sample Selection: As noted above, the first criteria used for the selection of clients at intake was that there was a CAFAS T1 (1 st Evaluation) on file. This criterion therefore eliminated all clients without such an evaluation from inclusion in the sample. For example, because the CAFAS is only used for clients from ages 6 to 17, this criterion eliminated clients aged 0-5 and 18 and over from the sample. Further, this criterion also eliminated clients who received services in only 1 to 3 sessions (as per CAFAS guidelines). c) Exit CAFAS (T14): There was some missing data from some of the CAFAS Exit profiles; these omissions are noted in the results section. d) BCFPI Parent Version: As noted above, this evaluation only considered the Parent Version of the BCFPI; Teacher and Youth versions were not used. In addition, not all clients included in the sample had a BCFPI Parent Version on file (In Care: 87.50% on file; Not In Care: 85.46% on file). Also, there were some items in some BCFPI s which had not been endorsed; these omissions are noted in the Results section. In reviewing and compiling the BCFPI data, it became evident that the BCFPI informants for the two groups were quite different. This is illustrated in Figure 5 below.

22 22 Figure 5 BCFPI Respondents % of Respondents Female Parent 5.36 Male Parent Female Parent 2 Unspecified Provider Unspecified Parent Step Father CAS Worker Foster Mother Foster Parent Missing In Care Not In Care For the children Not In Care, two informant types (Female Parent, Male Parent) made up for 95.43% of the total informants. Conversely, for the children In Care, it took 6 informant types (CAS Worker, Unspecified Provider, Female Parent 2, Female Parent, Male Parent, and Foster Mother) to constitute 95.19% of the total informants. e) Clients In Care: As noted above, once the sample was established, it was divided into two groups: clients who were in the care of a children s aid society, and those who were not in care of a children s aid society at the commencement of service. The evaluation did not consider: type of care (i.e. Crown Ward, Society Ward, Temporary Care, Customary Care, etc.) the length of time that the children had been in care whether the care status may have changed while the children were receiving service f) Services Provided: The evaluation did not consider the type, length or intensity of service provided, nor if service was completed (ex: drop out, moved away, etc.).

23 23 RESULTS CLIENTS AT INTAKE BCFPI scores were compared in two ways. First, the mean BCFPI subscale scores were compared. Second, the prevalence of Clinically scores on the BCFPI subscales was compared. Comparison of Means of BCFPI Scores on Each Subscale This method of comparison was used by the Ministry in preparing aggregate Northern Region Reports in the past (most recent report: Northern Region BCFPI Report, For the Quarter Ending June 2005). Means of Externalizing Mental Health Subscales The results are presented in Figure 6, and are described below. Percentile Ranks are presented as well. Percentile Ranks indicate that approximately x percent of the population scores below x (example: approximately 94.5% of the population score below a Percentile Rank of 94.5). Figure 6 BCFPI Means: Externalizing Scales Score Not Not Not 62.3 Not (p<.05) Regulating Attention Regulating Impulsivity and Activity Regulating Attention Impulsivity and Activity Cooperativeness With Others Conduct In Care Not In Care

24 24 Regulating Attention Group N Mean Percentile Rank In Care Not In Care * not significant Regulating Impulsivity and Activity Level* Group N Mean Percentile Rank In Care Not In Care * not significant Regulating Attention, Impulsivity and Activity Level Group N Mean Percentile Rank In Care Not In Care * not significant Cooperativeness with Others Group N Mean Percentile Rank In Care Not In Care * not significant Conduct* Group N Mean Percentile Rank In Care Not In Care * significant (p<.05) Means of Internalizing Mental Health Subscales The results are presented in Figure 7, and are described below.

25 25 Figure 7 BCFPI Means: Internalizing Scales Not Not (p<.05) Not Score Separating from Parents Managing Anxiety Managing Mood Self Harm In Care Not In Care Separating from Parents Group N Mean Percentile Rank In Care Not In Care * not significant Managing Anxiety Group N Mean Percentile Rank In Care Not In Care * not significant

26 26 Managing Mood* Group N Mean Percentile Rank In Care Not In Care * significant (p<.05) Self Harm Group N Mean Percentile Rank In Care Not In Care * not significant Means of Composite Scales Composite scores are more reliable measures of child functioning than individual subscales. They often constitute better estimates of overall risk, better measures of service outcome, and better predictors of the longer term course of child problems than individual subscales (Cunningham et al, October, 2006, p. 33).. The results are presented in Figure 8, and are described below.

27 27 Figure 8 BCFPI Means: Composite Scales Not Not Not Score Externalizing Behaviour Internalizing Behaviour Total Problems In Care Not In Care Externalizing Behaviour Group N Mean Percentile Rank In Care Not In Care * not significant Internalizing Behaviour Group N Mean Percentile Rank In Care Not In Care * not significant

28 28 Total Problems* Group N Mean Percentile Rank In Care Not In Care * not significant Means of Child Functional Impact Scales The Child Functional Impact Scales reflect the extent to which Mental Health Problems reviewed above are perceived to adversely affect the child's social participation, social relationships and academic performance. High t-scores on the Child Functional Impact Scale subscales reflect more severe functional impairment (Cunningham et al, October, 2006, p. 31). Children evidencing higher t-scores on the BCFPI's Child Functioning Scales are at higher risk than those with lower scores (Cunningham et al, October, 2006 p. 37). The results are presented in Figure 9 below. Figure 9 BCFPI Means: Child Functional Impact Scales (p<.05) (p<.05) (p<.05) 58.5 (p<.05) Score Social Participation Quality of Social Relationships School Participation and Achievement Impact on Child Functioning In Care Not In Care

29 29 Child' Social Participation Group N Mean Percentile Rank In Care Not In Care * significant (p less than.05) Quality of the Child's Social Relationships Group N Mean Percentile Rank In Care Not In Care * significant (p less than.05) School Participation and Achievement Group N Mean Percentile Rank In Care Not In Care * significant (p less than.05) Global Child/Youth Functioning Group N Mean Percentile Rank In Care Not In Care * significant (p less than.05) Means of Family Functional Impact Scales The results are presented in Figure 10, and are described below. The first two subscales (Family Activities, Family Comfort) describe the extent to which problems may be associated with a breakdown in family networks, conflict between partners, or overall distress regarding the child. These scores provide clues regarding issues which need to be addressed in follow-up assessments, potential targets for intervention, and family strengths (Cunningham et al, March, 2002, p. 28).

30 30 Figure 10 BCFPI Means: Family Functional Impact Scales Not (p<.05) 61.3 (p<.05) Score Family Activities Family Comfort Global Family Situation In Care Not In Care Family Activities Group N Mean Percentile Rank In Care Not In Care * not significant Family Comfort Group N Mean Percentile Rank In Care Not In Care * significant (p less than.05)

31 31 Global Family Situation: Group N Mean Percentile Rank In Care Not In Care * significant (p less than.05) Summary of Results of Comparison of BCFPI Means a) Are the average (mean) Brief Child and Family Phone Interview (BCFPI) subscale scores of children in care significantly different than those of children not in care? The results of the comparisons of means are summarized in Table 3 below. Table 3: BCFPI Subscale Means: Mean Subscale In Care Not In t-stat Care Regulating Attention No Regulating Impulsivity and Activity* No Regulating Attention Impulsivity &Activity No Cooperativeness No Conduct* Yes Separating from Parents No Managing Anxiety No Managing Mood* Yes Self Harm No Externalizing No Internalizing No Total Problems* No Social Participation Yes Quality of Relationships Yes School Participation and Achievement Yes Global Child/Youth Functioning Yes Family Activities No Family Comfort Yes Global Family Situation Yes Comparison of the means of the BCFPI Mental Health Subscales (9) and Composite Scales (3) revealed only 2 statistically significant differences on a total of 12 subscales. The In Care group had a higher mean score on the Conduct subscale, while the Not In Care group had a higher score on the Managing Mood subscale.